Showing posts with label flow. Show all posts
Showing posts with label flow. Show all posts

Tuesday, March 27, 2012

Data Source Views

I'm looking for examples of using Data Source Views as data source within a Data Flow. I have looked through the code examples supplied with the CTP but no joy.

Thanks
Al

You do not have to write code to use DSVs with Data Flows in SSIS.

You need to create a new Data Source and then a new DSV based on that Data Source.
Now add a new connection based on the Data Source to the Data Flow.
Add a OLEDB Source component and then you will have the choice of using the DSV in the OLEDB Source component UI.|||This will hopefully help

http://wiki.sqlis.com/default.aspx/SQLISWiki/DataSourceViewsAndSourceAdapters.html

Allan

"Alasdair Anderson@.discussions.microsoft.com" Anderson@.discussions.microsoft.com> wrote in message

news:88786eb8-c940-4590-b62a-485ae15b1803@.discussions.microsoft.com:

> I'm looking for examples of using Data Source Views as data source

> within a Data Flow. I have looked through the code examples supplied

> with the CTP but no joy.

>

> Thanks

> Al|||Very helpful guys thanks.

I have followup. How would you construct a data source view to look across data sources.

For example I have a Customers Dimension but the customer information sits across 3 hetrogeneous sources. Is it possible to sit a DSV across the 3 different databases to provide one view of customer to be used in the ETL?|||

Alasdair,
yes, this is possible. I don't know how to set it up specifically but I know you can definately do it.

-Jamie

Data Source Views

I'm looking for examples of using Data Source Views as data source within a Data Flow. I have looked through the code examples supplied with the CTP but no joy.

Thanks
Al

You do not have to write code to use DSVs with Data Flows in SSIS.

You need to create a new Data Source and then a new DSV based on that Data Source.
Now add a new connection based on the Data Source to the Data Flow.
Add a OLEDB Source component and then you will have the choice of using the DSV in the OLEDB Source component UI.|||This will hopefully help http://wiki.sqlis.com/default.aspx/SQLISWiki/DataSourceViewsAndSourceAdapters.html Allan "Alasdair Anderson@.discussions.microsoft.com" wrote in message news:88786eb8-c940-4590-b62a-485ae15b1803@.discussions.microsoft.com:
> I'm looking for examples of using Data Source Views as data source
> within a Data Flow. I have looked through the code examples supplied
> with the CTP but no joy. >
> Thanks
> Al|||Very helpful guys thanks.

I have followup. How would you construct a data source view to look across data sources.

For example I have a Customers Dimension but the customer information sits across 3 hetrogeneous sources. Is it possible to sit a DSV across the 3 different databases to provide one view of customer to be used in the ETL?|||

Alasdair,
yes, this is possible. I don't know how to set it up specifically but I know you can definately do it.

-Jamie

Tuesday, March 20, 2012

data reader source in data flow problem

hi all,

i have a package in ssis that needs to deliver data from outside servers with odbc connection. i have desined the package with dataflow object that includes inside a datareader source. the data reader source connect via ado.net odbc connection to the ouside servers and makes a query like: select * from x where y=? and then i pass the data to my sql server. my question is like the following:

how do i config the datasource reader or the dataflow so it will recognize an input value to my above query? i.e for example:

select * from x where y=5 (5 is a global variable that i have inside the package). i did not see anywhere where can i do it.

please help,

tomer

You can set an expression on the SQLCommand property. Similar to as explained here: http://blogs.conchango.com/jamiethomson/archive/2005/12/09/2480.aspx

The difference with the datareader source is that you cannot use a variable. Instead, set the expression on the SQLCommand property via the expressions of the parent data-flow.

-Jamie

|||

hi jamie,

thank you for your advice but i still did not understand completely.

the example u sent me is with ole db data source. lets say i have created the variable

and put there the select. now how do i config the data flow to use it?

can you give me another example?

thx,

Tomer

|||

Like I said, the Datareader Source does not allow you to use a variable as input. So, instead of putting the expression that builds your SQL statement into a variable, put it in an expression that sets the SQLCommand property of your Datareader Source. The interface to setting this expression is in the properties pane of the Data Flow task in which the Datareader Source resides.

-Jamie

|||

hi again Jamie,

i have put there:

"select * from x where y=" + @.[user::max_date]

and i get an error.

can u advice?

|||

Is max_date a Datetime variable? If so you need to cast it as a string in order to be able to concatenate it.

-Jamie

|||

Jamie,

thank you very much for your help - you realy helped me here and saved me lots of time.

tomer

|||

Hi Jamie,

yesterday everyyhing worked fine and suddenly i get in the package VS_BROKEN?

why can u advice me on that please?

tomer

|||

You have made a change in one of your components that means it won't work anymore. It should tell you which component.

-Jamie

|||

it says its failed executing the SQL command at the data reader source and i get validation error VS_BROKEN/ my sql is like yesterday:

"select * from x where y = " + [user::x]

when x is a string type. i must have changed somthing small without noticing...?

|||Are you missing the @. before the variable name (should be "@.[user::x]" ) in your sql command, or is that just a typo in your post?

Sunday, March 11, 2012

data not flowing out of flat file source

my package has a flat file source that should be extracting data from a text file passing the data to the next component in the data flow. the package validates fine, but the data isn't flowing. however, i see the data in the source component. i added a data viewer between the source and the next component to see if any data flowed and saw no data. can someone suggest how i should go about trying to debug this? thanks.Look at the output window. Anything of interest there?|||

Could you give more details about the flat file format and how you configured the flat file connection manager?

Thanks.

|||

DarrenSQLIS wrote:

Look at the output window. Anything of interest there?

below are some output lines that interest me:

Information: 0x40043007 at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Data Flow Lockbox Validate File and Header Info, Flat File Lockbox [1]: The processing of file "c:\casestudy\lockbox\samplelockbox.txt" has started.
Information: 0x400490F4 at Data Flow Lockbox Validate File and Header Info, Lookup BankBatchID [373]: component "Lookup BankBatchID" (373) has cached 0 rows.

"Flat File Lockbox [1]" is the flat file source component. the flat file connection manager is connected to file "c:\casestudy\lockbox\samplelockbox.txt"|||

Bob Bojanic wrote:

Could you give more details about the flat file format and how you configured the flat file connection manager?

Thanks.

below is the entire contents of the text file:

H080105 B1239-99Z-99 0058730760
I4001010003 181INTERNAT
C4001010004 01844400
I4002020005 151METROSPOOO1
C4002020006 02331800
I4003030009 MAGIC CYCLES
C4003030010 02697000
I4004040013 LINDELL
C4004040014 02131800
I4005040017 151GMASKI0001
C4005040019 01938800

the general tab of the flat file connection manager is configured as follows:

file name: c:\casestudy\lockbox\samplelockbox.txt
locale: english (united states)
unicode: unchecked
code page: 1252 (ANSI - Latin I)
format: ragged right
text qualifier: <none>
header row delimiter: {CR}{LF}
header rows to skip: 0
column names in first data row: unchecked|||

Can you try this?

In a copy of your package, delete everything after the flat file source.

Add a row count component after the Flat File Source (see http://msdn2.microsoft.com/en-us/library/ms141136(SQL.90).aspx for the row count component)

Now you can run the data flow with nothing but the flat file - if data flows to the row count, then your source is ok.

The reason I ask is becuase I see that your lookup component cached 0 rows - I wonder if that is actually the issue. Lookup row caching occurs on pre-execute so if there is a problem there, your flat file source will never even get started.

Donald Farmer

|||donald,

i took your suggestion and copied my package. then, i deleted everything from the data flow task except a flat file source and a rowcount. i then ran this package and the data still wasn't flowing out of the flat file source. next, i deleted my data flow task and created a new data flow task. then, i added a flat file source and a rowcount to this new data flow. low and behold, that resolved the issue. it seems that my original data flow somehow became corrupted and that this prevented the data from flowing out of the flat file source. to me, this seems to be a bug. is there a way to repair my original data flow so that it won't be necessary for me to duplicate all of my previous work?|||

Well that's odd, for sure. You can select, copy and paste components from data flow to data flow, so you could try copying and pasting the rest of your original data flow into your new one and hooking it up. There will be messages about metadata needing fixed up, but it all is effectively identical it should be relatively easy to do so.

(Copy and back up that new data flow first of course.)

I don't really have any suggestions about what could have gone wrong. I do wonder if the new data flow is identical in all ways to the old one, but difficult to tell without examining them in detail.

Donald

|||

Donald Farmer wrote:

Well that's odd, for sure. You can select, copy and paste components from data flow to data flow, so you could try copying and pasting the rest of your original data flow into your new one and hooking it up. There will be messages about metadata needing fixed up, but it all is effectively identical it should be relatively easy to do so.

(Copy and back up that new data flow first of course.)

I don't really have any suggestions about what could have gone wrong. I do wonder if the new data flow is identical in all ways to the old one, but difficult to tell without examining them in detail.

Donald

donald,

you were correct about there being messages about metadata needing fixed up. below are the messages:

TITLE: Package Validation Error

Package Validation Error

ADDITIONAL INFORMATION:

Error at Data Flow Task [DTS.Pipeline]: input column "line" (158) has lineage ID 28 that was not previously used in the Data Flow task.

Error at Data Flow Task [DTS.Pipeline]: "component "Derived Column Checks 1" (156)" failed validation and returned validation status "VS_NEEDSNEWMETADATA".

Error at Data Flow Task [DTS.Pipeline]: One or more component failed validation.

Error at Data Flow Task: There were errors during task validation.

(Microsoft.DataTransformationServices.VsIntegration)

you previously stated that fixing this should be relatively easy to do so. so, how should i go about fixing this?|||

There should be a warning triangle in the components that need to be fixed up. Double click on those components to open the UI and the metadata may be fixed automatically, or you will be prompted with a mapping dialog to fix up the changes.

Donald

|||

Donald Farmer wrote:

There should be a warning triangle in the components that need to be fixed up. Double click on those components to open the UI and the metadata may be fixed automatically, or you will be prompted with a mapping dialog to fix up the changes.

Donald

ok, that worked. thanks for your assistance.

Data Not Copying to MS Access Table

Hello,

I have a Data Flow Source that uses a SQL Command to pull data. In the SQL statement, I used CAST to change all varchar types to Nvarvchar to suit MS Access. I can preview the data from the source. In testing, the SQL statement only pulls about ten records.

I have a Microsoft 2000 Access database table as a destination. Data in each column in the table is required, and all columns have defaults.

I also have a grid data viewer set up. I have the DefaultBufferMaxRows set to 2 so that I can see data going across. When I execute this dataflow, no data is transfered to the Access database table. No data shows up in the dataviewer. There are no errors. The 'Execution Results' tab does not show errors, but indicates that zero rows were transfered. There are no warnings.

How do I begin to isolate the problem? The following is the SQL Statement in the Data Flow Source. Thank you for your help! - cdun2

DECLARE @.CategoryTable TABLE
(ColID Int,
ColCategory varchar(60),
ColValue varchar(500)
)

--and fill it

INSERT INTO @.CategoryTable
(ColID, ColCategory, ColValue)
SELECT
0,
LEFT(RawCollectionData,CHARINDEX(':',RawCollectionData)),
LTRIM(SUBSTRING(RawCollectionData,CHARINDEX(':',RawCollectionData)+1,255))
FROM Collections_Staging

--Assign an ID to each block of data for each occurance of 'Reason:'

DECLARE @.ID int
SET @.ID = 1
UPDATE @.CategoryTable
SET [ColID] = CASE WHEN ColCategory = 'Reason:' THEN @.ID - 1 ELSE @.ID END,
@.ID = CASE WHEN ColCategory = 'Reason:' THEN @.ID + 1 ELSE @.ID END

--Then put the data together

SELECT --cast to Nvarchar for MSAccess
a.ColID,
CAST(a.ColValue as Nvarchar(30)) AS OrderID,
COALESCE(CAST(b.ColValue as Nvarchar(30)),'') AS SellerUserID,
COALESCE(CAST(c.ColValue as Nvarchar(100)),'') AS BusinessName,
COALESCE(CAST(d.ColValue as Nvarchar(15)),'') AS BankID,
COALESCE(CAST(e.ColValue as Nvarchar(15)),'') AS AccountID,
COALESCE(CAST(SUBSTRING(f.ColValue,CHARINDEX('$',f.ColValue)+1,500)AS DECIMAL(18,2)),0) AS CollectionAmount,
COALESCE(CAST(g.ColValue as Nvarchar(10)),'') AS TransactionType,
CASE
WHEN h.ColValue LIKE '%Matching Disbursement%' THEN NULL
ELSE CAST(h.ColValue AS SmallDateTime)
END AS DisbursementDate,
--COALESCE(h.ColValue,'') AS DisbursementDate,
CASE
WHEN i.ColValue LIKE '%Matching Disbursements%' THEN NULL
WHEN CAST(LEFT(REVERSE(i.ColValue),4)AS INT) > 1000 THEN CAST(i.ColValue AS SmallDateTime)
WHEN LEFT(REVERSE(i.ColValue),4) = '1000' THEN NULL
END AS ReturnDate,
--COALESCE(i.ColValue,'') AS ReturnDate,
COALESCE(CAST(j.ColValue as Nvarchar(4)),'') AS Code,
COALESCE(CAST(k.ColValue as Nvarchar(255)),'') AS CollectionReason
FROM @.CategoryTable a
LEFT JOIN @.CategoryTable b ON b.ColID = a.ColID AND b.ColCategory = 'Seller UserId:'
LEFT JOIN @.CategoryTable c ON c.ColID = a.ColID AND c.ColCategory = 'Business Name:'
LEFT JOIN @.CategoryTable d ON d.ColID = a.ColID AND d.ColCategory = 'Bank ID:'
LEFT JOIN @.CategoryTable e ON e.ColID = a.ColID AND e.ColCategory = 'Account ID:'
LEFT JOIN @.CategoryTable f ON f.ColID = a.ColID AND f.ColCategory = 'Amount:'
LEFT JOIN @.CategoryTable g ON g.ColID = a.ColID AND g.ColCategory = 'Transaction Type:'
LEFT JOIN @.CategoryTable h ON h.ColID = a.ColID AND h.ColCategory = 'Disbursement Date:'
LEFT JOIN @.CategoryTable i ON i.ColID = a.ColID AND i.ColCategory = 'Return Date:'
LEFT JOIN @.CategoryTable j ON j.ColID = a.ColID AND j.ColCategory = 'Code:'
LEFT JOIN @.CategoryTable k ON k.ColID = a.ColID AND k.ColCategory = 'Reason:'

WHERE a.ColCategory = 'Order ID:'

Are you doing this on an x64 machine. Is it possible that your package is executed in 64-bit mode? There is no 64-bit version of the JET provider.

There have been many posts about this before. Seaarch this forum to get instructions for making sure the packages are executed in 32-bit mode.

Thanks,

Bob

|||

Hi Cdun,

When you preview the Source, do you find any rows existing for the query? If yes, When you execute the dataflow task can you see any rows in Data grid viewer?

In fact if there are rows, it must be transfered to the destination table.

Thanks

Subhash Subramanyam

|||

Subhash Subramanyam wrote:

When you execute the dataflow task can you see any rows in Data grid viewer?

There are rows in the preview, but no rows in the data grid viewer. I'll check into the 64 bit setting.

|||

Bob Bojanic - MSFT wrote:

Seaarch this forum to get instructions for making sure the packages are executed in 32-bit mode.

Thanks,

Bob

I went to the package properties, debugging, and set Run64BitRuntime to False. I still get the same result The data will preview, but will not transfer into the Access table.

In the Data Flow Task, I'm using an OLEDB source to execute the sql statement. Should I be using an Execute SQL Task instead? The sql statement also uses a TABLE variable. Could this be a problem?

|||There must be something wrong with the way I'm trying to deliver the data to the Access table, because I can't get data to a sql server table destination either.|||

Never mind on this. I resorted to creating a table UDF as the source, and it looks like that will work.

cdun2

data not appearing

In the data flow, I have an "OLE DB source" container that calls an SP. The SP does a select statement at the end. If I hit the "Preview" button I'm seeing the results come back, however, when I go to the columns I'm not seeing anything. I don't see the column name. Is there any reason why I don't seem to see this column coming back?

Thanks,

Phil

That issues has been address in this fourm. Try searching..I just found these:

http://forums.microsoft.com/MSDN/Search/Search.aspx?words=OLE+SOurce+stored+procedure&localechoice=9&SiteID=1&searchscope=forumscope&ForumID=80

Tuesday, February 14, 2012

Data Flow: Converting data in multiple columns

Hi,

I'm just wondering what's the best approach in Data Flow to convert the following input file format:

Date, Code1, Value1, Code2, Value2

1-Jan-2006, abc1, 20.00, xyz3, 35.00

2-Jan-2006, abc1, 30.00, xyz5, 6.30

into the following output format (to be loaded into a SQL DB):

Date, Code, Value

1-Jan-2006, abc1, 20.00

1-Jan-2006, xyz3, 35.00

2-Jan-2006, abc1, 30.00

2-Jan-2006, xyz5, 6.30

I'm quite new to SSIS, so, I would appreciate detailed steps if possible. Thanks.

Ok, I have found a method to get what I wanted, but I'm not sure if it's the best approach. Any comments appreciated.

I first used a Multicast and feed the input data into 2 Script Components. The first Script Component has input columns of Date, Code1 & Value1, while 2nd Script Component has input columns of Date, Code2 & Value2. In both Script Components, the output columns are Date, Code & Value.

The output of each Script Component are then connected to a Sort and both Sort goes to a Merge. The output from the Merge will then go to a OLE DB Destination to be loaded into a SQL DB (not 2005 version).

Hope it makes sense.

|||

I think Unpivot tranformation cam deliver the same funcionality you put in your script components; but the multicast; the sort and merge join would be required anyway.

Rafael Salas

|||

Hi,

Why dont you have a union step feeded twice by the multicast

then

Delete and add lines so that you would have the following

Date date date

Field 2 cod1 cod2

Field 3 value1 value2

At the end you should have what you want,

Regards,

data flow,

Lets say I have 10 tasks in the data flow, but I want to execute only 5 tasks which don't depend on any other tasks in that data flow.
I don't want to execute all my tasks all the time while I am in development stage. Is it possible to execute that way? How can I do that?

Before answering can I just clarify something.

You can't put tasks into a data-flow so do you mean:
-Tasks in the control-flow
or
-Components in a data-flow

?

-Jamie|||Sorry for the confusion. Components in a data flow task.
So I have a data flow task created in Control flow and then I have
10 components (OLE DB Src and OLE DB dest connections) that extracts data and loads data in the Data Flow. I want to execute 2 out of those 10 to load my new 2 tables.

|||You can't stop tasks running in a dataflow. You can only limit how many sources are processing at the same time. So if you had 10 sources and only wanted 2 at a time you could set the EngineThreads property to 2 and only 2 would be going at any one time but all would get executed before the dataflow completed. If you don't want this then you would need to create separate dataflows for the ones you want to run together and then disable the dataflows you didn't want to run.

HTH,
Matt

data flow with fuzzy lookup freezes on pre-execute phase

I have a fuzzy lookup in a data flow but this data flow never gets pass pre-execute phase. What is the problem?

Thanks

does the package execute up to that point just fine? how is the fuzzy lookup configured, what is the size of the dataset you're trying to load up?

|||when i excute the whole the whole package , it works fine, not much delay, however, running the data flow task by itselfs hung

Data Flow Tasks - SSIS

I am using Recordset destination for SSIS. Flat file destination will
not work and Excel destination can not handle more that 65k rows and I
don't want to send my output to the server. Therefore after doing a
Merge join I want to view the results and Recordset seemed to be a
valid choice. The problem is how do I view the output? For other
components sometimes there are preview option. However this preview is
not consistent for all steps in the data flow tasks and sometimes
missing. My question is there any way to preview the Recordset
destination? Thanks.On May 25, 11:38 am, SB <othell...@.yahoo.com> wrote:
> I am using Recordset destination for SSIS. Flat file destination will
> not work and Excel destination can not handle more that 65k rows and I
> don't want to send my output to the server. Therefore after doing a
> Merge join I want to view the results and Recordset seemed to be a
> valid choice. The problem is how do I view the output? For other
> components sometimes there are preview option. However this preview is
> not consistent for all steps in the data flow tasks and sometimes
> missing. My question is there any way to preview the Recordset
> destination? Thanks.
Okay, instead of Recordset I have used DataReaderDestination and
created a Grid viewer to view the data. The problem is, it is very
slow and hangs after processing 96k out of 396k of records. (This is
one simple update involving a 2 table join that I can do in seconds in
the server) It is taking hours to develop the package and when it did
come back it is only showing 9k out of 396k of data in the grid. How
do I see all the data? I have tried attach/detach but still can't seem
to view all the data! How do I view all the data in the grid? Thanks.

Data Flow Tasks - SSIS

I am using Recordset destination for SSIS. Flat file destination will
not work and Excel destination can not handle more that 65k rows and I
don't want to send my output to the server. Therefore after doing a
Merge join I want to view the results and Recordset seemed to be a
valid choice. The problem is how do I view the output? For other
components sometimes there are preview option. However this preview is
not consistent for all steps in the data flow tasks and sometimes
missing. My question is there any way to preview the Recordset
destination? Thanks.
On May 25, 11:38 am, SB <othell...@.yahoo.com> wrote:
> I am using Recordset destination for SSIS. Flat file destination will
> not work and Excel destination can not handle more that 65k rows and I
> don't want to send my output to the server. Therefore after doing a
> Merge join I want to view the results and Recordset seemed to be a
> valid choice. The problem is how do I view the output? For other
> components sometimes there are preview option. However this preview is
> not consistent for all steps in the data flow tasks and sometimes
> missing. My question is there any way to preview the Recordset
> destination? Thanks.
Okay, instead of Recordset I have used DataReaderDestination and
created a Grid viewer to view the data. The problem is, it is very
slow and hangs after processing 96k out of 396k of records. (This is
one simple update involving a 2 table join that I can do in seconds in
the server) It is taking hours to develop the package and when it did
come back it is only showing 9k out of 396k of data in the grid. How
do I see all the data? I have tried attach/detach but still can't seem
to view all the data! How do I view all the data in the grid? Thanks.

Data Flow Tasks - SSIS

I am using Recordset destination for SSIS. Flat file destination will
not work and Excel destination can not handle more that 65k rows and I
don't want to send my output to the server. Therefore after doing a
Merge join I want to view the results and Recordset seemed to be a
valid choice. The problem is how do I view the output? For other
components sometimes there are preview option. However this preview is
not consistent for all steps in the data flow tasks and sometimes
missing. My question is there any way to preview the Recordset
destination? Thanks.On May 25, 11:38 am, SB <othell...@.yahoo.com> wrote:
> I am using Recordset destination for SSIS. Flat file destination will
> not work and Excel destination can not handle more that 65k rows and I
> don't want to send my output to the server. Therefore after doing a
> Merge join I want to view the results and Recordset seemed to be a
> valid choice. The problem is how do I view the output? For other
> components sometimes there are preview option. However this preview is
> not consistent for all steps in the data flow tasks and sometimes
> missing. My question is there any way to preview the Recordset
> destination? Thanks.
Okay, instead of Recordset I have used DataReaderDestination and
created a Grid viewer to view the data. The problem is, it is very
slow and hangs after processing 96k out of 396k of records. (This is
one simple update involving a 2 table join that I can do in seconds in
the server) It is taking hours to develop the package and when it did
come back it is only showing 9k out of 396k of data in the grid. How
do I see all the data? I have tried attach/detach but still can't seem
to view all the data! How do I view all the data in the grid? Thanks.

Data Flow task within For Each Loop not executing

All:

I am sure I am missing something really silly but I am not able to figure out what. The For Each Loop uses an ADO Enumerator and passes variable values to a data flow. In executing the package the loop runs fine but nothing is happening to the data flow. When I move the data flow out of the loop it runs fine. What is going on?

Thanks!

desibull

desibull wrote:

All:

I am sure I am missing something really silly but I am not able to figure out what. The For Each Loop uses an ADO Enumerator and passes variable values to a data flow. In executing the package the loop runs fine but nothing is happening to the data flow. When I move the data flow out of the loop it runs fine. What is going on?

Thanks!

desibull

If the dataflow uses variables that are passed to it via the ForEach loop, how does the dataflow work when it is moved out of the ForEach loop? Do the variables have package scope?

Regards

|||

Yes, the variables have package scope.

It is really odd. The control does not seem to be flowing to the data flow. It jsut sits there doing nothing while the loop finishes executing.

|||

desibull wrote:

Yes, the variables have package scope.

It is really odd. The control does not seem to be flowing to the data flow. It jsut sits there doing nothing while the loop finishes executing.


Strange. Try dragging another executable (I suggest an empty sequence container) into the ForEach loop and see if it executes.

-Jamie

|||

I am such a dodo! Basically there were no records to select. I change a variable and there were rows to select and now everything runs fine.

What is still odd is that even when there were no records to select the data flow still showed the yellow and green colors when places outside the loop but remained white while within the loop, and that is why I panicked.

Thanks again!

|||

desibull wrote:

I am such a dodo! Basically there were no records to select. I change a variable and there were rows to select and now everything runs fine.

What is still odd is that even when there were no records to select the data flow still showed the yellow and green colors when places outside the loop but remained white while within the loop, and that is why I panicked.

Thanks again!

Strange. They should still change colour. Can I suggest you raise this at connect.microsoft.com with a repro?

-Jamie

|||Still being a newbie to this forum could you explain what repro means and what is expected to be posted to connect? Is it a visual display of the flow?|||

desibull wrote:

Still being a newbie to this forum could you explain what repro means and what is expected to be posted to connect? Is it a visual display of the flow?

Sorry. "repro" means reproduction. i.e. Something that demonstrates teh problem and can be executed by Microsoft.


Connect is a place for posting what you think is a bug. You should post anything that helps explain the problem. Words, pictures, repro, whatever.

-Jamie

|||

Jamie Thomson wrote:

desibull wrote:

I am such a dodo! Basically there were no records to select. I change a variable and there were rows to select and now everything runs fine.

What is still odd is that even when there were no records to select the data flow still showed the yellow and green colors when places outside the loop but remained white while within the loop, and that is why I panicked.

Thanks again!

Strange. They should still change colour. Can I suggest you raise this at connect.microsoft.com with a repro?

-Jamie

I have a different viewpoint on this. Since the data flow task is never executing (there are no records, so the For Each loop isn't execute anything inside it), it shouldn't show green (or yellow, or red).

|||

jwelch wrote:

I have a different viewpoint on this. Since the data flow task is never executing (there are no records, so the For Each loop isn't execute anything inside it), it shouldn't show green (or yellow, or red).

John,

The dataflow would still have to execute in order for it to KNOW that there were no records. The same is true of the source adapters. Hence, if a source adapter is executing then it must create at least one buffer - even if the buffer is empty. If a buffer is created then I would expect the other components to process that buffer.

That's how I figure it in my head anyway.

-Jamie

|||

I agree, if the data flow is actually being run. If I am interpreting the OP's comments correctly, though, the Data Flow isn't executing because there were no rows retrieved in the ADO resultset that is driving the For Each loop, . If there are rows in the resultset that is used by the For Each, then the data flow should be executing and it should show green. If, however, there are no rows in the For Each resultset, it shouldn't be executing the data flow at all, so I would expect it to remain white.

|||

jwelch wrote:

I agree, if the data flow is actually being run. If I am interpreting the OP's comments correctly, though, the Data Flow isn't executing because there were no rows retrieved in the ADO resultset that is driving the For Each loop, . If there are rows in the resultset that is used by the For Each, then the data flow should be executing and it should show green. If, however, there are no rows in the For Each resultset, it shouldn't be executing the data flow at all, so I would expect it to remain white.

Ah OK. I'll guess we'll have to see if he replies to clarify Smile

|||

John is right on the money. I observed that the data flow within the ForEach loop does not execute when the ADO Recordset driving the loop is empty.

But I am still unable to understand why the data flow will not execute at least once because the loop executes once using the default values of the variables. Now, using the default values the data flow does not pull any records but it should at least execute, right?

Anyways, my observation confirms John's viewpoint.

Thank you!

|||

There is no concept of default values. Yes, there are values stored there at design-time but they won't get used if you using another method to set them at execution-time.

-Jamie

Data Flow task within For Each Loop not executing

All:

I am sure I am missing something really silly but I am not able to figure out what. The For Each Loop uses an ADO Enumerator and passes variable values to a data flow. In executing the package the loop runs fine but nothing is happening to the data flow. When I move the data flow out of the loop it runs fine. What is going on?

Thanks!

desibull

desibull wrote:

All:

I am sure I am missing something really silly but I am not able to figure out what. The For Each Loop uses an ADO Enumerator and passes variable values to a data flow. In executing the package the loop runs fine but nothing is happening to the data flow. When I move the data flow out of the loop it runs fine. What is going on?

Thanks!

desibull

If the dataflow uses variables that are passed to it via the ForEach loop, how does the dataflow work when it is moved out of the ForEach loop? Do the variables have package scope?

Regards

|||

Yes, the variables have package scope.

It is really odd. The control does not seem to be flowing to the data flow. It jsut sits there doing nothing while the loop finishes executing.

|||

desibull wrote:

Yes, the variables have package scope.

It is really odd. The control does not seem to be flowing to the data flow. It jsut sits there doing nothing while the loop finishes executing.


Strange. Try dragging another executable (I suggest an empty sequence container) into the ForEach loop and see if it executes.

-Jamie

|||

I am such a dodo! Basically there were no records to select. I change a variable and there were rows to select and now everything runs fine.

What is still odd is that even when there were no records to select the data flow still showed the yellow and green colors when places outside the loop but remained white while within the loop, and that is why I panicked.

Thanks again!

|||

desibull wrote:

I am such a dodo! Basically there were no records to select. I change a variable and there were rows to select and now everything runs fine.

What is still odd is that even when there were no records to select the data flow still showed the yellow and green colors when places outside the loop but remained white while within the loop, and that is why I panicked.

Thanks again!

Strange. They should still change colour. Can I suggest you raise this at connect.microsoft.com with a repro?

-Jamie

|||Still being a newbie to this forum could you explain what repro means and what is expected to be posted to connect? Is it a visual display of the flow?|||

desibull wrote:

Still being a newbie to this forum could you explain what repro means and what is expected to be posted to connect? Is it a visual display of the flow?

Sorry. "repro" means reproduction. i.e. Something that demonstrates teh problem and can be executed by Microsoft.


Connect is a place for posting what you think is a bug. You should post anything that helps explain the problem. Words, pictures, repro, whatever.

-Jamie

|||

Jamie Thomson wrote:

desibull wrote:

I am such a dodo! Basically there were no records to select. I change a variable and there were rows to select and now everything runs fine.

What is still odd is that even when there were no records to select the data flow still showed the yellow and green colors when places outside the loop but remained white while within the loop, and that is why I panicked.

Thanks again!

Strange. They should still change colour. Can I suggest you raise this at connect.microsoft.com with a repro?

-Jamie

I have a different viewpoint on this. Since the data flow task is never executing (there are no records, so the For Each loop isn't execute anything inside it), it shouldn't show green (or yellow, or red).

|||

jwelch wrote:

I have a different viewpoint on this. Since the data flow task is never executing (there are no records, so the For Each loop isn't execute anything inside it), it shouldn't show green (or yellow, or red).

John,

The dataflow would still have to execute in order for it to KNOW that there were no records. The same is true of the source adapters. Hence, if a source adapter is executing then it must create at least one buffer - even if the buffer is empty. If a buffer is created then I would expect the other components to process that buffer.

That's how I figure it in my head anyway.

-Jamie

|||

I agree, if the data flow is actually being run. If I am interpreting the OP's comments correctly, though, the Data Flow isn't executing because there were no rows retrieved in the ADO resultset that is driving the For Each loop, . If there are rows in the resultset that is used by the For Each, then the data flow should be executing and it should show green. If, however, there are no rows in the For Each resultset, it shouldn't be executing the data flow at all, so I would expect it to remain white.

|||

jwelch wrote:

I agree, if the data flow is actually being run. If I am interpreting the OP's comments correctly, though, the Data Flow isn't executing because there were no rows retrieved in the ADO resultset that is driving the For Each loop, . If there are rows in the resultset that is used by the For Each, then the data flow should be executing and it should show green. If, however, there are no rows in the For Each resultset, it shouldn't be executing the data flow at all, so I would expect it to remain white.

Ah OK. I'll guess we'll have to see if he replies to clarify Smile

|||

John is right on the money. I observed that the data flow within the ForEach loop does not execute when the ADO Recordset driving the loop is empty.

But I am still unable to understand why the data flow will not execute at least once because the loop executes once using the default values of the variables. Now, using the default values the data flow does not pull any records but it should at least execute, right?

Anyways, my observation confirms John's viewpoint.

Thank you!

|||

There is no concept of default values. Yes, there are values stored there at design-time but they won't get used if you using another method to set them at execution-time.

-Jamie

Data flow task to delete records and then insert records in transaction

HI,

I have been trying to solve the locking problem from past couple of days. Please help mee!!

Scenario:
--
I have a SSIS package in which 2 data flow tasks. 1st data flow task deletes records from a 5 tables and the 2nd data flow task should insert records into 1 of the five tables after the success of 1st data flow task. This scenario runs in Transacation.

The above scenrio in the 2nd data flow task hangs in runtime. It does not complete. with sp_who2 command i could see that there is an intent share lock(LK_M_IS) on the table and the status is SUSPENDED.

I dont know how to come out of this locking. Please help.

Thanks ,
SunilTry setting RetainSameConnection to TRUE on the connection manager.

|||It was already set to TRUE. Please help meeee..

|||Based on some other threads about similar issues, this may not be SSIS, but related to DTC and the core relational engine. You might try checking some of the other forums to see if they have any suggestions.

|||

Have you tried using a SQL Task (set based) for the deletes followed by your Insert Data Flow Task?

Larry

|||

Hey Larry,

Yes. I did use it.but still the blocking happens. Please suggest

Thanks,

Sunil

Data Flow Task Stuck

I have a simple data flow task setup...

2 ADO.NET connection managers, each referencing a DSN pointed to a Unidata database.

2 DataReader sources, each using a single ADO.NET connection managers, running a simple SELECT statement from a table.

I have a Union All transform setup to merge the data and write to a OLE DB Destination (SQL05 database)


When I run the package, each source will validate, but only one will execute. The other source will do nothing. The data source will be colored yellow, and will just sit there. The package will just sit, almost like it is waiting for input.


This behavior is not consistent, however. It varies which data source will hang, pretty much 50-50. About 25% of the time, both sources will execute, and all rows will be written to the destination.


Any help is appreciated.


thanks



Are the source connections pointing to the same database? I'm not familiar with Unidata, but it sounds like there might be some type of blocking issue, either in the database or in the ODBC driver. If you use two separate data flows, one per DataReader Source, does it consistently run successfully?|||No, they are not pointing to the same database, and I have tried independent data flows with the same results.



|||

When you tried independent data flows, did you add a precedence constraint so that they were executing sequentially rather than in parallel?

If the problem shows up when you are only running a single DataReader Source, I'd see if there are any known issues with the ODBC driver.

|||When I try them sequentially, it works correctly every time.


This is an acceptable workaround, but I am still curious to determine the reason for this strange behavio

|||

Corey M. wrote:

When I try them sequentially, it works correctly every time.


This is an acceptable workaround, but I am still curious to determine the reason for this strange behavio

My guess is that it's an ODBC driver issue. Perhaps table locking or something like that. Who knows. You'll have to check for any known issues with the ODBC driver provider.

|||

You may wish to try the flowsync transform, placing one FlowSync transform between each source adapter and the union all transform (for a total of two FlowSync transforms).

The FlowSync transform is a speed governor, so a particular source flow doesn't get too far ahead of the other one. For something like union all, one source outpacing another should not be an issue, but you could certainly give it a shot.

To install the transform, you'll need to copy the assembly (FlowSync.dll) into two places and add it to the toolbar.

1. For runtime purposes, copy the FlowSync.dll into the global assembly cache, which is "%windir%\assembly" or typically the directory c:\windows\assembly. For design-time purposes, copy the same FlowSync.dll into the Integration Service PipelineComponents directory, which is typically located at "%ProgramFiles%\Microsoft SQL Server\90\DTS\PipelineComponents", If you're running 64 bit, the design-time installation directory is typically located at "%ProgramFiles(x86)%\Microsoft SQL Server\90\DTS\PipelineComponents".

2. To add flowsync transform to the toolbar, open up BIDS, right click on the toolbox, select "Choose Items...", go to the tab SSIS Data Flow Items, and Select the FlowSync transform, which will then show up appear in the toolbox as an available transform.

Data Flow Task SQL strings

Hi,

I just wanna ask:

I'm creating an SSIS package, a Data Flow Task. I have used OLEDB Source connected to a SQL Server Destination. Now in my OLEDB Source, I have this SQL statement

SELECT FirstName, LastName, Age FROM Employees WHERE (Age > 10) AND (Age < 95)

But what I want is to have the last name and first name concatenated and in proper case(capitalize first letter of the firstname and surname). I also want to TRIM or remove the blank spaces of the field in my SQL statement. How I be able to do this?

I tried using proper(), trim() and ucase() like in MSAccess but no success.

Please help. Thanks in advance.

The OLE DB Source has to use the same syntax as used by the underlying DB engine - in this case SQL Server.

Have a look in BOL for RTRIM(), LTRIM(), SUBSTRING(), UPPER() LEFT(), REPLACE() to get you going.

Alternatively you could carry out this work in the SSIS data-flow using a Derived Column component.

-Jamie

|||Thanks for your reply. I was able to do it using Derived Column. Thanks again and more power...

Data flow task running very slow

Hello,

I developed an SSIS package doing a nightly load into a data warehouse. We have an 8 hour loading window - currently the package takes 16 hours to complete.

I isolated the problem to a Data Flow task where +-35% of the time is spent. This task is pretty straight forward:

- OLE DB source, reading +- 800,000 rows from a SQL server database

- 13 Lookups in sequence, to get surrogate keys from dimension tables. Lookups are all on GUIDS.

- An aggregation

- OLEDB target, fact table in a SQL server database.

It seems unreasonable for the this task to take over 5 hours. It spends the majority of time on the lookups - not so much at target, source and aggregation.

Any comments and advice will be greatly appreciated.

Thanks.

(PS some machine details:

OS Name Microsoft(R) Windows(R) Server 2003, Standard Edition
Version 5.2.3790 Service Pack 1 Build 3790
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name ARK-SQL
System Manufacturer HP
System Model ProLiant DL380 G5
System Type X86-based PC
Processor x86 Family 6 Model 15 Stepping 6 GenuineIntel ~1866 Mhz
Processor x86 Family 6 Model 15 Stepping 6 GenuineIntel ~1866 Mhz
BIOS Version/Date HP P56, 9/18/2006
SMBIOS Version 2.3
Windows Directory C:\WINDOWS
System Directory C:\WINDOWS\system32
Boot Device \Device\HarddiskVolume1
Locale United States
Hardware Abstraction Layer Version = "5.2.3790.1830 (srv03_sp1_rtm.050324-1447)"
User Name Not Available
Time Zone South Africa Standard Time
Total Physical Memory 3,327.30 MB
Available Physical Memory 938.20 MB
Total Virtual Memory 1.10 GB
Available Virtual Memory 2.78 GB
Page File Space 2.00 GB
Page File C:\pagefile.sys)

How many rows are the lookups caching? Are you selecting the whole table (All columns) or are you using a SQL query to specify only the columns you need.

Are you sure it is not your dest that is slow receiving the rows?
If it is, SSIS will slow down the rows retrieved from the source making it appear as if it is something else slowing it down.

Check the lookups. Are as few columns as possible being selected.
Push your rows into nothing such as a Konesans Trash Destination. Does it appear faster?

Also, if your lookup is caching alot of rows and the key you are caching is a GUID, that's a chunk of work to do.|||

You need to find where the bottleneck is. You could start measuring how fast the dataflow 'reads from source'; then how fast it does the transformations (since you have a fair amount of transformations, I would measure a several points); and finally measure the how fast it writes into the destination.

few tips:

The lookups could slow down performance if you are using partial cache. So, limit the number of columns and rows (by providing a select statement with a where clause if possible) and use full cache mode. This approach could generate another problem if memory in server is limited.

See if you can move the aggregation upstream and or limit the number of columns to be used as 'group by'. In general aggregations will perform better if the number of columns with fewer columns in the 'group by'.

If you use an OLE DB destination, try to stick with 'fast load' and set the interval commit to an acceptable range

Use DB profiler tools to measure the performance of each SQL statement use in the data flow (OLE DB source, lookups, etc)

This white paper has some other information

http://www.microsoft.com/technet/prodtechnol/sql/2005/ssisperf.mspx|||

I would also highly recommend taking a read of this:

Donald Farmer's Technet webcast

(http://blogs.conchango.com/jamiethomson/archive/2006/06/14/SSIS_3A00_-Donald-Farmer_2700_s-Technet-webcast.aspx)

-Jamie

|||

Hi Crispin,

Thanks for your response.

I solved the problem last night. The key was in the caching.

I modify the SQL statement in most of my lookups to handle inferred fact entries. I.e. a sale against an unknown customer gets linked to the 'UNKNOWN CUTOMER' dimension table entry. It seems that SSIS sets the caching to partial when you modify the lookup SQL. I assume the lookup had to go back to disk quite often and therefore high cost for big dimension tables (...of which I have quite a few).

I am achieving a +-500% performance increase by keeping the Lookup transform as it is and by setting the cahcing to full. I treat the inferred fact entries at a later stage in stead of at the point of lookup.

|||Good to hear it has approved, however...

The lookup does not change to partial cache when you specify a statement. Not sure why you say it has done this. Was the checkbox checked?

On the partial cache thing, you are correct. The lookup will check it's cache, if the key is found, use it, else run the query against SQL. This can become dog slow. Only time it has been a good thing, for me at least, is when you know your facts are going to be a very small portion of a really large dimension. That way, the first few queries are slow but you don't have to cache 13 million rows only to use 1000.

Data flow task reports different row count than actual rowcount

I have a data flow task that moves all the rows from 18 tables on a production server to a reporting services server. One table, which does not contain the most rows (about 650K rows) reports all the rows have been transferred. However, if I go in to the SQL Mgmt Studio and do a Select count(*) on the table, there are only 110k rows.

Has anyone else experienced this problem?

Thanks,

Nick Anzano

How does the data flow report that 650k rows have been transferred? Are you looking in the logs?

SSIS could certainly report that it has sent 650k to the database - what happens then is up to the database! Is the target database SQL Server? If so, use SQL Profiler to see what is happening when the rows are being sent to the server. Perhaps they are not being committed for some reason.

Donald Farmer

Data Flow Task question

Hi there. I'm trying to learn SSIS, please, help me. I have 2 questions:

1)
There are 2 databases on 2 different servers. I need to get data from Table1(database1) and put it to Table2(database2). But I have to insert rows, which ID is not exists in Table2. How Can I do necessary filter?

2)
In the OLE DB DataSource Component I have used SQL Command(it's simplified):

declare @.TmpTable TABLE (WorkCode int not null);

INSERT INTO @.TmpTable (WorkCode)
select WorkCode
from Table1

SELECT WorkCode
FROM @.TmpTable

SSIS Package works without any exception. But there is no any inserted record in destination table. If I try similar query without temporary table - it works good. Why?

1 - Create OLE DB source to 1st (source) database.
- Add a lookup transformation to select key from 2nd database on 2nd server. In that lookup join the key coming from the 1st database to the key in the 2nd
- Hook the error output (red arrow) from the lookup to an OLE DB destination which points to the 2nd database. This will insert records not found in the 2nd database.

2 - Try setting the RetainSameConnection property of the connection manager to true and see what happens.|||The first task works! Thanks! But the second is not. Any other ideas?
|||

Aliaksander Hmyrak wrote:

The first task works! Thanks! But the second is not. Any other ideas?

Good deal.

As for number 2, why are you using temp tables? 2 things - you can just write the query that inserts into the temp table as the source for the data flow. OR you could use yet another, initial, data flow to populate a SQL server table with the results you need in the 2nd data flow (the one you've already got written). After the two data flows, you could write an Execute SQL task in the control flow to truncate that "temporary" table.|||

At the start of your SQL statement add

Code Snippet

SET NOCOUNT ON

|||Thanks a lot !! Both methods work!