Showing posts with label approach. Show all posts
Showing posts with label approach. Show all posts

Tuesday, March 27, 2012

Data Source passwords in Config Files

Anybody find a sound approach to being able to store DataSource connection information in a configuration and be able to access a Data Source that requires a password for login?

Storing it as a configuraiton in the DB is not an option, as you require the pw to access the DB that would store it.

Can security be applied to an environment variable? Registry?

Anyone successfully use the registry to store configuration values?I'm not sure what your criterion for sound is, so I don't know if this meets it :), but you can store credentials (username & password) in a configuration file, and then make an environment variable which gives its path, and then configure a Package Configuration in the SSIS package to use the environment variable.
Of course you have to configure the environment variable for each machine (and for each user, if two users on one machine will use different credentials -- therefore, different configuration files).
Then, the credentials being now in one or more files, secure access to these files via traditional NTFS ACLs (file pemissions).
|||

I guess by sound, I meant, any way that works... when i store the pw in the config the .xml file represents it like the following:

- <Configuration ConfiguredType="Property" Path="\Package.Connections [MDB].Properties[Password]" ValueType="String">

<ConfiguredValue />

</Configuration>

This results in the error:
Error 1 Validation error. Data Flow Task: OLE DB Source [1]: The AcquireConnection method call to the connection manager "MDB" failed with error code 0xC0202009. Package2.dtsx 0 0

Have you been able to do this?.. am I missing something? I am also storing the connection string, initial catalog, and username in the config file.

|||When the file is generated by Visual Studio, it will not write the pw. I was able to get it to work by modifying the file directly.. duh.|||I'm assuming that you're not on a domain and so cannot use integrated authentication?
K|||Yes, Joe, We've been using the solution I explained above, with different configurations for different developers on the same machine (so they can use different login names, so the processes are distinguishable in SQL Server Enterprise Manager).
But, we've been using SQL Server logons (not integrated), because we're using a SQL Server 2000 backend. What you quoted sounds like you're using an mdb backend, which we've not tried.
To set up a new configuration with a new set of credentials, I first add the environment variable for that user, pointing it to where the config file is going to be. Then I put a config file there -- I take an existing one, and using a text editor, change the logon name & password (stored in plaintext in the file).
The very first time we did this, SSIS generated the config file for us; since then, we've been hand editing it for each machine and/or credential combination.

|||Here is an edited copy of our config file. Everything beginning with Zz is something that I edited before posting (along with the nifty DEADBEEF guid).
Also I added the carriage returns and whitespaces -- the raw config file is a whitespaceless mess.
******************

<?xml version="1.0"?>
<DTSConfiguration>
<DTSConfigurationHeading>
<DTSConfigurationFileInfo GeneratedBy="ZzMachineName\ZzUserName" GeneratedFromPackageName="Zz Our Nifty Package" GeneratedFromPackageID="{DEADBEEF-AAAA-BBBB-CCCC-DEADBEEFBEEF}" GeneratedDate="7/26/2005 4:25:01 PM"/>
</DTSConfigurationHeading>
<Configuration ConfiguredType="Property" Path="\Package.Connections[ZZ_OUR_CONN_NAME].Properties[Password]" ValueType="String">
<ConfiguredValue>ZzBigSecretPassword</ConfiguredValue>
</Configuration>
<Configuration ConfiguredType="Property" Path="\Package.Connections[ZZ_OUR_CONN_NAME].Properties[UserName]" ValueType="String">
<ConfiguredValue>ZzLogonNameForThisDeveloperHere</ConfiguredValue>
</Configuration>
<Configuration ConfiguredType="Property" Path="\Package.Connections[ZZ_OUR_NOTHER_CONN].Properties[Password]" ValueType="String">
<ConfiguredValue>ZzBigSecretPassword</ConfiguredValue>
</Configuration>
<Configuration ConfiguredType="Property" Path="\Package.Connections[ZZ_OUR_NOTHER_CONN].Properties[UserName]" ValueType="String">
<ConfiguredValue>ZzLogonNameForThisDeveloperHere</ConfiguredValue>
</Configuration>
</DTSConfiguration>

Sunday, March 11, 2012

Data Normalization - Best approach?

Hi!

I wonder what would be the best (at to be honest - how to do it at all) to perform data normalization with SSIS. The scenario is as follows:

    I got plain table with several columns in it.Some of columns can be copied straight into destination tableSome columns (String) should be lookup in another table to get ID
      On success just replace string with IDOn fail - create new record in lookup table and return newly created ID
Thanks for any ideas and maybe short samples

Anrijs Vitolins

Yeah you can do all of this. This link will get you started regarding the lookup and putting new records into the lookup table - I recommend method 2: http://www.sqlis.com/default.aspx?311

You'll need a MULTICAST transform in there at some point as well by the sounds of it.

-Jamie

Data Modeling for FACTS with Different Granularity

What is the best approach in data modeling FACTS with
different granularity?
For instance, I have Volume measures (item count, item
amount) at daily intervals, but Cost measures (Processing
costs, service costs for those items) at monthly and
quarterly intervals.
How best to construct the data model with FACT table(s)
able to query calculated measures like "daily unit cost"?
Is it inevitable that I would have to separate the volume
counts and cumulative costs into 2 separate FACT tables?
I am looking for ideas how to best to consolidate volume
and costs into ONE FACT table. I would imagine this is a
common situation where the volume info are not coming at
the same interval and aggregate as your costs info (vendor
sends invoices monthly or quarterly). So, please chime in
your thoughts.
Thanks in advance!
This is what Analysis Services does for you behind the scenes -- It creates
data structures with whatever levels of aggregation you require so that
real-time aggregation doesn't need to take place when you query the data.
If you're using SQL Server only, you should look into using indexed views,
which SQL Server can use when appropriate, automatically (you do not need to
explicitely reference them in your queries).
You should not model different levels of aggregation in the same table, in
my opinion, as it creates logical ambiguities (the table will no longer
represent a single entity type).
"Steven Wong" <anonymous@.discussions.microsoft.com> wrote in message
news:ae6701c436b0$1139a130$a101280a@.phx.gbl...
> What is the best approach in data modeling FACTS with
> different granularity?
> For instance, I have Volume measures (item count, item
> amount) at daily intervals, but Cost measures (Processing
> costs, service costs for those items) at monthly and
> quarterly intervals.
> How best to construct the data model with FACT table(s)
> able to query calculated measures like "daily unit cost"?
> Is it inevitable that I would have to separate the volume
> counts and cumulative costs into 2 separate FACT tables?
> I am looking for ideas how to best to consolidate volume
> and costs into ONE FACT table. I would imagine this is a
> common situation where the volume info are not coming at
> the same interval and aggregate as your costs info (vendor
> sends invoices monthly or quarterly). So, please chime in
> your thoughts.
> Thanks in advance!
|||Hi :
When u have 2 different fact tables with different granularities, the best
choice is to use them separatelly. The other way u should get into the
allocation problem. Remember u can connect them using 2 conformed dimensions
(Shared Dims). If u need help, just let me know
ALEJANDRO LEGUIZAMO
Colombia - SQL Server MVP
"Steven Wong" <anonymous@.discussions.microsoft.com> wrote in message
news:ae6701c436b0$1139a130$a101280a@.phx.gbl...
> What is the best approach in data modeling FACTS with
> different granularity?
> For instance, I have Volume measures (item count, item
> amount) at daily intervals, but Cost measures (Processing
> costs, service costs for those items) at monthly and
> quarterly intervals.
> How best to construct the data model with FACT table(s)
> able to query calculated measures like "daily unit cost"?
> Is it inevitable that I would have to separate the volume
> counts and cumulative costs into 2 separate FACT tables?
> I am looking for ideas how to best to consolidate volume
> and costs into ONE FACT table. I would imagine this is a
> common situation where the volume info are not coming at
> the same interval and aggregate as your costs info (vendor
> sends invoices monthly or quarterly). So, please chime in
> your thoughts.
> Thanks in advance!
|||The following assumes that the Cost fact tables include unit cost
information. If not, you would have to derive them as something like last
know unit cost.
It sounds like you would need at least two fact tables: a monthly cost fact
table, a quarterly cost fact table, and the volume fact table. Populate the
two cost fact tables first, then reference them in your ETL for the volume
fact table. Add measures to your volume fact table for latest known unit
costs.
It is not a good idea to ever try to mix granularities in a fact table. You
either need to aggregate to a common level, or allocate down to a rate (as I
did above).
"Steven Wong" <anonymous@.discussions.microsoft.com> wrote in message
news:ae6701c436b0$1139a130$a101280a@.phx.gbl...
> What is the best approach in data modeling FACTS with
> different granularity?
> For instance, I have Volume measures (item count, item
> amount) at daily intervals, but Cost measures (Processing
> costs, service costs for those items) at monthly and
> quarterly intervals.
> How best to construct the data model with FACT table(s)
> able to query calculated measures like "daily unit cost"?
> Is it inevitable that I would have to separate the volume
> counts and cumulative costs into 2 separate FACT tables?
> I am looking for ideas how to best to consolidate volume
> and costs into ONE FACT table. I would imagine this is a
> common situation where the volume info are not coming at
> the same interval and aggregate as your costs info (vendor
> sends invoices monthly or quarterly). So, please chime in
> your thoughts.
> Thanks in advance!

Data Modeling for FACTS with Different Granularity

What is the best approach in data modeling FACTS with
different granularity?
For instance, I have Volume measures (item count, item
amount) at daily intervals, but Cost measures (Processing
costs, service costs for those items) at monthly and
quarterly intervals.
How best to construct the data model with FACT table(s)
able to query calculated measures like "daily unit cost"?
Is it inevitable that I would have to separate the volume
counts and cumulative costs into 2 separate FACT tables?
I am looking for ideas how to best to consolidate volume
and costs into ONE FACT table. I would imagine this is a
common situation where the volume info are not coming at
the same interval and aggregate as your costs info (vendor
sends invoices monthly or quarterly). So, please chime in
your thoughts.
Thanks in advance!This is what Analysis Services does for you behind the scenes -- It creates
data structures with whatever levels of aggregation you require so that
real-time aggregation doesn't need to take place when you query the data.
If you're using SQL Server only, you should look into using indexed views,
which SQL Server can use when appropriate, automatically (you do not need to
explicitely reference them in your queries).
You should not model different levels of aggregation in the same table, in
my opinion, as it creates logical ambiguities (the table will no longer
represent a single entity type).
"Steven Wong" <anonymous@.discussions.microsoft.com> wrote in message
news:ae6701c436b0$1139a130$a101280a@.phx.gbl...
> What is the best approach in data modeling FACTS with
> different granularity?
> For instance, I have Volume measures (item count, item
> amount) at daily intervals, but Cost measures (Processing
> costs, service costs for those items) at monthly and
> quarterly intervals.
> How best to construct the data model with FACT table(s)
> able to query calculated measures like "daily unit cost"?
> Is it inevitable that I would have to separate the volume
> counts and cumulative costs into 2 separate FACT tables?
> I am looking for ideas how to best to consolidate volume
> and costs into ONE FACT table. I would imagine this is a
> common situation where the volume info are not coming at
> the same interval and aggregate as your costs info (vendor
> sends invoices monthly or quarterly). So, please chime in
> your thoughts.
> Thanks in advance!|||Hi :
When u have 2 different fact tables with different granularities, the best
choice is to use them separatelly. The other way u should get into the
allocation problem. Remember u can connect them using 2 conformed dimensions
(Shared Dims). If u need help, just let me know
ALEJANDRO LEGUIZAMO
Colombia - SQL Server MVP
"Steven Wong" <anonymous@.discussions.microsoft.com> wrote in message
news:ae6701c436b0$1139a130$a101280a@.phx.gbl...
> What is the best approach in data modeling FACTS with
> different granularity?
> For instance, I have Volume measures (item count, item
> amount) at daily intervals, but Cost measures (Processing
> costs, service costs for those items) at monthly and
> quarterly intervals.
> How best to construct the data model with FACT table(s)
> able to query calculated measures like "daily unit cost"?
> Is it inevitable that I would have to separate the volume
> counts and cumulative costs into 2 separate FACT tables?
> I am looking for ideas how to best to consolidate volume
> and costs into ONE FACT table. I would imagine this is a
> common situation where the volume info are not coming at
> the same interval and aggregate as your costs info (vendor
> sends invoices monthly or quarterly). So, please chime in
> your thoughts.
> Thanks in advance!|||The following assumes that the Cost fact tables include unit cost
information. If not, you would have to derive them as something like last
know unit cost.
It sounds like you would need at least two fact tables: a monthly cost fact
table, a quarterly cost fact table, and the volume fact table. Populate the
two cost fact tables first, then reference them in your ETL for the volume
fact table. Add measures to your volume fact table for latest known unit
costs.
It is not a good idea to ever try to mix granularities in a fact table. You
either need to aggregate to a common level, or allocate down to a rate (as I
did above).
"Steven Wong" <anonymous@.discussions.microsoft.com> wrote in message
news:ae6701c436b0$1139a130$a101280a@.phx.gbl...
> What is the best approach in data modeling FACTS with
> different granularity?
> For instance, I have Volume measures (item count, item
> amount) at daily intervals, but Cost measures (Processing
> costs, service costs for those items) at monthly and
> quarterly intervals.
> How best to construct the data model with FACT table(s)
> able to query calculated measures like "daily unit cost"?
> Is it inevitable that I would have to separate the volume
> counts and cumulative costs into 2 separate FACT tables?
> I am looking for ideas how to best to consolidate volume
> and costs into ONE FACT table. I would imagine this is a
> common situation where the volume info are not coming at
> the same interval and aggregate as your costs info (vendor
> sends invoices monthly or quarterly). So, please chime in
> your thoughts.
> Thanks in advance!

Tuesday, February 14, 2012

Data Flow: Converting data in multiple columns

Hi,

I'm just wondering what's the best approach in Data Flow to convert the following input file format:

Date, Code1, Value1, Code2, Value2

1-Jan-2006, abc1, 20.00, xyz3, 35.00

2-Jan-2006, abc1, 30.00, xyz5, 6.30

into the following output format (to be loaded into a SQL DB):

Date, Code, Value

1-Jan-2006, abc1, 20.00

1-Jan-2006, xyz3, 35.00

2-Jan-2006, abc1, 30.00

2-Jan-2006, xyz5, 6.30

I'm quite new to SSIS, so, I would appreciate detailed steps if possible. Thanks.

Ok, I have found a method to get what I wanted, but I'm not sure if it's the best approach. Any comments appreciated.

I first used a Multicast and feed the input data into 2 Script Components. The first Script Component has input columns of Date, Code1 & Value1, while 2nd Script Component has input columns of Date, Code2 & Value2. In both Script Components, the output columns are Date, Code & Value.

The output of each Script Component are then connected to a Sort and both Sort goes to a Merge. The output from the Merge will then go to a OLE DB Destination to be loaded into a SQL DB (not 2005 version).

Hope it makes sense.

|||

I think Unpivot tranformation cam deliver the same funcionality you put in your script components; but the multicast; the sort and merge join would be required anyway.

Rafael Salas

|||

Hi,

Why dont you have a union step feeded twice by the multicast

then

Delete and add lines so that you would have the following

Date date date

Field 2 cod1 cod2

Field 3 value1 value2

At the end you should have what you want,

Regards,