Showing posts with label package. Show all posts
Showing posts with label package. Show all posts

Wednesday, March 28, 2012

Poor performance import text file

Hi,

I'm very new at SSIS. I've got a package to import a 1,000,000 row tab delimited text file into my SQL2005 database. Using SSIS with a Text File source and OLE DB PROVIDER target it crawls. I do have it skipping error rows but that is the only logic I am using. No indexes on the table either. This is pretty basic and it takes SSIS 1.5 hours, DTS blows this away... Assuming this is not something outside of SSIS(like the network, or a user locking issue), what I am I doing wrong?

BTW -- I'm searching for some batch commit sizes, like in DTS, but I cannot find them.

Any ideas.

From the information that you've given, it's hard to know what the problem is, but here are a few things you can check.

Try to isolate the problem. Change the destination to a rowcount or trash destination from SQLIS.com, see if it improves the throughput.

Is the flatfile on the same machine as the executing package?

Are you minimally logged on the destination?

Is the destination DB on the same machine?

Have you tried the SQL Server Destination?

How many errors are you getting from the flat file?

Rows per batch is on the OLEDB Destination Adapter. Did you try changing that?

K

Monday, March 26, 2012

Poll For a File

Hi All,

I am using ActiveX code in a DTS package (Package1) to search a directory for a file.
When the file is there I want to start a DTS package (Package2) to load
in this file.

I will schedule Package1 to run at 10pm, however since the external file is being generated externally I cannot guarentee that the file will be there at 10pm. Therefore I want to Poll the folder and not start package2 until I have the file. I am using the If file exists method.

How do I poll for a file, and if its not there wait for 5 mins and poll again until I find the file.

Thanks in advanceHave you tried FSO objects?

Dim objFSO
Set objFSO = CreateObject("Scripting.FileSystemObject")|||I have tried the FSO and thats what I am using.

My problem is that if the file is not there, I want to wait for a period of
time ie 5 mins and then try again. And keep trying every five mins until I get the file.

Some thing like this:

========================
Function Main()

dim result

set objFSO = CreateObject("Scripting.FileSystemObject")

result = 1

DO WHILE result = 1
If objFSO.FileExists("c:\test.txt") Then
//Start other Package
result = 99
Else
wait(5)
//Go to start and check for file again
End if

LOOP

End Function
=========================

So in essence the above code will LOOP until it finds the file.
I just need something to replace the wait(5)

Any Ideas

Thanks in advance

I have wait(5) but wait is not a valid function|||EXEC [master].[dbo].[xp_fileexist] 'c:\new_auth.dat1'

This might help|||Originally posted by superquinn
I have tried the FSO and thats what I am using.

My problem is that if the file is not there, I want to wait for a period of
time ie 5 mins and then try again. And keep trying every five mins until I get the file.

Some thing like this:

========================
Function Main()

dim result

set objFSO = CreateObject("Scripting.FileSystemObject")

result = 1

DO WHILE result = 1
If objFSO.FileExists("c:\test.txt") Then
//Start other Package
result = 99
Else
wait(5)
//Go to start and check for file again
End if

LOOP

End Function
=========================

So in essence the above code will LOOP until it finds the file.
I just need something to replace the wait(5)

Any Ideas

Thanks in advance

I have wait(5) but wait is not a valid function

Why do not create a new function (I did not check it, sorry) and call it:

Function wait(theDate)
Do Until DateDiff("n", Now, theDate)<5
Loop
End Function|||I dont want to write a loop function like you have suggested as this constant looping might be very hard on the CPU. I was hoping there was a clever way or else an in built wait/sleep function available to me in the ActiveX component of the DTS.

Can Anyone help?|||You could write a Agent Job that will run every 5 minutes from 10 to 12. Sounds like the easiest way.|||Hi,

Im not quiet sure I understand what you mean by an agent Job.

I am only new to SQL Server and DTS.

The polling for a file is causing a lot of trouble for me.
Can someone please help|||In enterprise Manager, expand Management, SQL Server Agent, then click on Jobs. From there you can add a new job, which can execute ActiveX scripts, T-SQL scripts, even DTS packages. You can add one or many schedules to kick off the job.

I think it may be what you are looking for.

You can get started by right-clicking on the DTS package you created, and selecting Schedule Package, which will set the DTS package up to run as a job. You can then adjust the schedule to your liking, add more steps, etc.|||Would the command WAITFOR DELAY be useful?

Here is the help from MS SQL Server Books Online...
WAITFOR
Specifies a time, time interval, or event that triggers the execution of a statement block, stored procedure, or transaction.

Syntax
WAITFOR { DELAY 'time' | TIME 'time' }

Arguments
DELAY

Instructs Microsoft SQL Server to wait until the specified amount of time has passed, up to a maximum of 24 hours.

'time'

Is the amount of time to wait. time can be specified in one of the acceptable formats for datetime data, or it can be specified as a local variable. Dates cannot be specified; therefore, the date portion of the datetime value is not allowed.

TIME

Instructs SQL Server to wait until the specified time.

Remarks
After executing the WAITFOR statement, you cannot use your connection to SQL Server until the time or event that you specified occurs.

To see the active and waiting processes, use sp_who.

Examples
A. Use WAITFOR TIME
This example executes the stored procedure update_all_stats at 10:20 P.M.

BEGIN
WAITFOR TIME '22:20'
EXECUTE update_all_stats
END

For more information about using this procedure to update all statistics for a database, see the examples in UPDATE STATISTICS.

B. Use WAITFOR DELAY
This example shows how a local variable can be used with the WAITFOR DELAY option. A stored procedure is created to wait for a variable amount of time and then returns information to the user as to the number of hours, minutes, and seconds that have elapsed.

CREATE PROCEDURE time_delay @.@.DELAYLENGTH char(9)
AS
DECLARE @.@.RETURNINFO varchar(255)
BEGIN
WAITFOR DELAY @.@.DELAYLENGTH
SELECT @.@.RETURNINFO = 'A total time of ' +
SUBSTRING(@.@.DELAYLENGTH, 1, 3) +
' hours, ' +
SUBSTRING(@.@.DELAYLENGTH, 5, 2) +
' minutes, and ' +
SUBSTRING(@.@.DELAYLENGTH, 8, 2) +
' seconds, ' +
'has elapsed! Your time is up.'
PRINT @.@.RETURNINFO
END
GO
-- This next statement executes the time_delay procedure.
EXEC time_delay '000:00:10'
GO

Here is the result set:

A total time of 000 hours, 00 minutes, and 10 seconds, has elapsed! Your time is up.|||Hi,

Thanks for your reply. I dont think I am explaining myself fully

I know how to sechdule the Package.
I know how to check if file exists

What I need to know is how to halt processing in a activex script
for a period of time.

If the file is not there initially, I want the wait a while and the reloop to
the start of the package and so on until the file is there

==========ACTIVEX SCRIPT========

Set foundfile = 0

Do While foundfile = 0

if foundfile = 1 then
set foundfile = 1 (This breaks loop)
//Carry on processing
Set Success
else
//Halt procesing for 5 mins and then continue
//This is where I need HELP!!

end if

loop|||I think we understand what you are asking, but are offering solutions that differ from the model you are chasing.

For instance, when I mentioned a scheduled job, I meant that you could create a task that would check for the file's existence. If were not there, the job would schedule itself to run again in a few minutes, then exit. If it were there, proceed to the next step, which would kick off your DTS package.|||The problem with having an ActiveX script wait, is that you are tying up a thread and processor resources just counting away clock cycles. The SQL Agent dooes that already, so you might as well take advantage of it.|||Originally posted by bpdWork
I think we understand what you are asking, but are offering solutions that differ from the model you are chasing.

For instance, when I mentioned a scheduled job, I meant that you could create a task that would check for the file's existence. If were not there, the job would schedule itself to run again in a few minutes, then exit. If it were there, proceed to the next step, which would kick off your DTS package.

=====================================
This sounds EXACTLY like what I want to do. Im sorry about this
but Im new to SQL Server.

How do you get a Job to reschedule itself in code?
I assume this is done in ActiveX|||Off the top of my head, I would say that you would need to use an ActiveX script to access the SQLDMO object, and control it from there. I have done such a thing before. I will see what I can find in my old code heap.|||I think I can help

You should create your package wiht the activex script ect...
Then schedule it to run every 5 mins. as a SQL job.

The active x script should look like this. No need to make the activex script wait, just have it run again and again. If no file, it just quits, when it finds one it runs the other import package.

Function Main()

Dim objFSO
Dim objFolder
Dim objFile
Dim oPKG
Dim fileName
Dim folderName

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFolder = objFSO.GetFolder("\\server\share$\dir\")

For Each objFile in objFolder.Files

fileName = objFile.Path

Set oPKG = CreateObject("DTS.Package")
oPKG.LoadFromSQLServer "Server", "sa", "", , , , , "2nd package name here"
oPKG.GlobalVariables("fileName").Value = fileName
oPKG.Execute
oPKG.Uninitialize()
Set oPKG = Nothing

objFSO.MoveFile fileName, folderName 'move file somewhere
Next

Main = DTSTaskExecResult_Success

End Function

Hope this helps

Steve|||yes thats the easiest way,

The first step in your package will be a activeX task with the following script inside

==========================================
Function Main()
Dim fs, MyFile
Dim FileLocation
Dim FileName
Dim FileInfo

FileName = "file.txt"

FileLocation = DTSGlobalVariables("Invoice_Input_Location").Value

FileInfo = FileLocation & FileName

Set fs = CreateObject("Scripting.FileSystemObject")

If (fs.FileExists(FileInfo)) Then
fs.MoveFile (FileInfo),(FileLocation & "new_file.txt")
set fs=nothing
Main = DTSTaskExecResult_Success
exit function
Else
Main = DTSTaskExecResult_Failure
exit function
End If

End Function
==========================================

rest of the steps inside the DTS will be after success of this task. The package will be schedules to run after every 5 mins, if your file is present, it will be renamed with a different name (so that next time when the package runs afetr 5 mins it does not process it all over again), this renamed file can be used for further transformations. This way the package will run successfully just once everyday and will fail for rest of the times. which is what you want.

Monday, March 12, 2012

Pls Help - can't install Sql Server Reporting Services

I have Sql Server Personal Edition SP3 installed on my PC. I get an error
message when trying to install the reporting services trial package. Does
the reporting services not work with a personal edition of Sql Server?
The error message is "The edition of the report server database is not
supported on the edition of Sql Server 2000 you have chosen".
Thanks,Paul, I will be interested to see any responses you receive I have been
experiencing the same problems for the past 2 days. With what appear to be
the exact same setup you are running XP SP2, SQL 2000 SP3a and trial edition
of reporting services.
"Paul Tilson" wrote:
> I have Sql Server Personal Edition SP3 installed on my PC. I get an error
> message when trying to install the reporting services trial package. Does
> the reporting services not work with a personal edition of Sql Server?
> The error message is "The edition of the report server database is not
> supported on the edition of Sql Server 2000 you have chosen".
> Thanks,
>
>|||The System Requirements page at
http://www.microsoft.com/sql/reporting/productinfo/sysreqs.asp says that
only Standard, Enterprise, and Developer Editions are supported. If you're
just doing this for trial purposes, how about installing the Evaluation
Edition of SQL Server while you're testing?
--
Sincerely,
Stephen Dybing
This posting is provided "AS IS" with no warranties, and confers no rights.
"Paul Tilson" <tilsonpa@.yahoo.com> wrote in message
news:10o3egfkodqaic4@.corp.supernews.com...
>I have Sql Server Personal Edition SP3 installed on my PC. I get an error
>message when trying to install the reporting services trial package. Does
>the reporting services not work with a personal edition of Sql Server?
> The error message is "The edition of the report server database is not
> supported on the edition of Sql Server 2000 you have chosen".
> Thanks,
>

Monday, February 20, 2012

Please help! Suspended/Stalled Query

Hi,

I am having a problem with one package that attempts to do a join on two large tables. Within the data flow task control is the ole db source control which has the initial query that joins and filters the tables.

The total number of rows that should be returned are about 80-90million. Running this in debug mode shows the number of rows read/inserted into the ole db destination and it always hits the about the same number of records where it just stops/stalls/suspends around 75 million. I have read some possible solutions such as changing the max degrees of parrallelism but that has not worked.

I am currently using SQL Server 2005 64bit SP2 on a 64 bit platform. This is the only package that does anything like this and I am at my wit's end with this problem.

Please help!

Thanks,
Cheston

Can you share some more information about what's in your data flow? Sorts, aggregates, OLE DB Commands, etc...

Also, what happens when you run this without debugging?

|||

SELECT FCT_POST_MTHLY_FINANCE_REVENUE.POST_MTHLY_FINANCE_REVENUE_KEY, FCT_POST_MTHLY_FINANCE_REVENUE.SUBSCRIBER_KEY,
FCT_POST_MTHLY_FINANCE_REVENUE.MSF_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.SAF_REVENUE,
FCT_POST_MTHLY_FINANCE_REVENUE.AIRTIME_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.LD_INTL_REVENUE,
FCT_POST_MTHLY_FINANCE_REVENUE.LOAD_TS, FCT_POST_MTHLY_FINANCE_REVENUE.ROAMING_REVENUE,
FCT_POST_MTHLY_FINANCE_REVENUE.LD_CAN_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.BILL_MONTH_KEY,
FCT_POST_MTHLY_FINANCE_REVENUE.DATA_ON_VOICE_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.UPDATE_TS,
FCT_POST_MTHLY_FINANCE_REVENUE.OPTIONAL_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.MISCELLANEOUS_REVENUE,
FCT_POST_MTHLY_FINANCE_REVENUE.SIF_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.LD_US_REVENUE,
FCT_POST_MTHLY_FINANCE_REVENUE.CONSOLIDATED_REVENUE, FCT_POST_MTHLY_FINANCE_REVENUE.ARPU_TOTAL_REVENUE,
FCT_POST_MTHLY_FINANCE_REVENUE.UNCLASSIFIED_REVENUE
FROM FCT_POST_MTHLY_REVENUE_NEW RIGHT OUTER JOIN
FCT_POST_MTHLY_FINANCE_REVENUE ON
FCT_POST_MTHLY_REVENUE_NEW.BILL_MONTH_KEY = FCT_POST_MTHLY_FINANCE_REVENUE.BILL_MONTH_KEY AND
FCT_POST_MTHLY_REVENUE_NEW.SUBSCRIBER_KEY = FCT_POST_MTHLY_FINANCE_REVENUE.SUBSCRIBER_KEY
WHERE (FCT_POST_MTHLY_REVENUE_NEW.SUBSCRIBER_KEY IS NULL)

This is my query that I am trying to run. Actually all I am currently trying to do is a row count. It gets suspended there as well at the same number as when I was trying to direct the data into a table.

BTW, it stops exactly at 75, 627, 900

|||

So basically you have the OLE DB source and simply a row count transformation and it hangs? (Just the two components on the data flow?)

Have you made sure that your query isn't timing out, or network related? SQL Profiler can help here.

Also, I'm still curious to know what happens when you run this without debugging.

|||

Yes, there are only 2 components to the data flow.

I do not believe it the query timing out or network related but I will investigate into that. Also will try running it without debugging and seeing if it works.

|||

Audit Logout Microsoft SQL Server Management Studio 0 0 0 340000 3644 53 2007-03-10 17:55:21.153 2007-03-10 18:01:01.153
Audit Login -- network protocol: LPC
set quoted_identifier on
set arithabort off
set numeric_roundabort off
set ansi_warnings on
set ansi_padding on
set ansi_nulls on
set concat_null_yields_null on
set cursor_close_on_commit off
set implicit_transactions off
set language us_english
set dateformat mdy
set datefirst 7
set transaction isolation level read committed
Microsoft SQL Server Management Studio Administrator 3644 53 2007-03-10 18:01:01.153

Hey Phil,

I did the trace on the package with just the row count. The above is what keeps repeating once I hit the 75, 627, 900 records. I am unfamiliar with what this is, can you explain?

|||quick bump|||From what you describe it could be a deadlock situation.

Look at the Activity Monitor and see WHY it is stalled.