Hi,
we have cubes partitioned on the time dimension. The majority of data that
comes in if for the last 4 days or so. The new data is loaded into a seperate
fact table, and loaded into a partition. This partition is then merged into
the current month partition.
However, we can have data that comes in which is a month or longer old. So
would be merged into the incorrect partition. And would remain there until a
full reprocess of the Fact tables are cube partitiions.
I understand that when data slices are defined for a partition, that AS2000
will only visit the partitions that contain the relevent data. What happens
if we use data slices in our situation?
If the data is in the incorrect partition, as a result of merging, is it
possible that the data is missed out, depending on the query? Or would the
merging not be allowed to take place?
Would we want to define data slices for all partitions, except for the
current month ?
Or would we want to define a filter, and rather than a build and merge, do
an incremental build against the cube ?
We are really looking for the quickest load time possible, while having
correct data of course.
Thanks in advance for any advice and help.
Is this for AS 2000 or 2005? In 2000, slices that you defined are used for
querying. In 2005, slices for MOLAP partitions are detected automatically.
For ROLAP partitions the slice you specify will be used to eliminate
partitions from queries.
But besides that, only your latest partition would have "incorrect" data,
right? So just don't set a slice for that last partition -- it would mean
that this partition would be unnecessarily scanned for some queries, but it
should be a fairly small partition anyway...
Thanks,
Akshai
--
This posting is provided "AS IS" with no warranties, and confers no rights
Please do not send email directly to this alias. This alias is for newsgroup
purposes only.
"Al" <Al@.discussions.microsoft.com> wrote in message
news:370E22EC-6304-49F2-B01D-812B37B2E1EF@.microsoft.com...
> Hi,
> we have cubes partitioned on the time dimension. The majority of data that
> comes in if for the last 4 days or so. The new data is loaded into a
> seperate
> fact table, and loaded into a partition. This partition is then merged
> into
> the current month partition.
> However, we can have data that comes in which is a month or longer old. So
> would be merged into the incorrect partition. And would remain there until
> a
> full reprocess of the Fact tables are cube partitiions.
> I understand that when data slices are defined for a partition, that
> AS2000
> will only visit the partitions that contain the relevent data. What
> happens
> if we use data slices in our situation?
> If the data is in the incorrect partition, as a result of merging, is it
> possible that the data is missed out, depending on the query? Or would the
> merging not be allowed to take place?
> Would we want to define data slices for all partitions, except for the
> current month ?
> Or would we want to define a filter, and rather than a build and merge, do
> an incremental build against the cube ?
> We are really looking for the quickest load time possible, while having
> correct data of course.
> Thanks in advance for any advice and help.
>
Showing posts with label seperate. Show all posts
Showing posts with label seperate. Show all posts
Sunday, March 25, 2012
Data Sliles or Filter or seperate Fact Tables for ourCube Partitio
Hi,
we have cubes partitioned on the time dimension. The majority of data that
comes in if for the last 4 days or so. The new data is loaded into a seperat
e
fact table, and loaded into a partition. This partition is then merged into
the current month partition.
However, we can have data that comes in which is a month or longer old. So
would be merged into the incorrect partition. And would remain there until a
full reprocess of the Fact tables are cube partitiions.
I understand that when data slices are defined for a partition, that AS2000
will only visit the partitions that contain the relevent data. What happens
if we use data slices in our situation?
If the data is in the incorrect partition, as a result of merging, is it
possible that the data is missed out, depending on the query? Or would the
merging not be allowed to take place?
Would we want to define data slices for all partitions, except for the
current month ?
Or would we want to define a filter, and rather than a build and merge, do
an incremental build against the cube ?
We are really looking for the quickest load time possible, while having
correct data of course.
Thanks in advance for any advice and help.Is this for AS 2000 or 2005? In 2000, slices that you defined are used for
querying. In 2005, slices for MOLAP partitions are detected automatically.
For ROLAP partitions the slice you specify will be used to eliminate
partitions from queries.
But besides that, only your latest partition would have "incorrect" data,
right? So just don't set a slice for that last partition -- it would mean
that this partition would be unnecessarily scanned for some queries, but it
should be a fairly small partition anyway...
Thanks,
Akshai
--
This posting is provided "AS IS" with no warranties, and confers no rights
Please do not send email directly to this alias. This alias is for newsgroup
purposes only.
"Al" <Al@.discussions.microsoft.com> wrote in message
news:370E22EC-6304-49F2-B01D-812B37B2E1EF@.microsoft.com...
> Hi,
> we have cubes partitioned on the time dimension. The majority of data that
> comes in if for the last 4 days or so. The new data is loaded into a
> seperate
> fact table, and loaded into a partition. This partition is then merged
> into
> the current month partition.
> However, we can have data that comes in which is a month or longer old. So
> would be merged into the incorrect partition. And would remain there until
> a
> full reprocess of the Fact tables are cube partitiions.
> I understand that when data slices are defined for a partition, that
> AS2000
> will only visit the partitions that contain the relevent data. What
> happens
> if we use data slices in our situation?
> If the data is in the incorrect partition, as a result of merging, is it
> possible that the data is missed out, depending on the query? Or would the
> merging not be allowed to take place?
> Would we want to define data slices for all partitions, except for the
> current month ?
> Or would we want to define a filter, and rather than a build and merge, do
> an incremental build against the cube ?
> We are really looking for the quickest load time possible, while having
> correct data of course.
> Thanks in advance for any advice and help.
>sql
we have cubes partitioned on the time dimension. The majority of data that
comes in if for the last 4 days or so. The new data is loaded into a seperat
e
fact table, and loaded into a partition. This partition is then merged into
the current month partition.
However, we can have data that comes in which is a month or longer old. So
would be merged into the incorrect partition. And would remain there until a
full reprocess of the Fact tables are cube partitiions.
I understand that when data slices are defined for a partition, that AS2000
will only visit the partitions that contain the relevent data. What happens
if we use data slices in our situation?
If the data is in the incorrect partition, as a result of merging, is it
possible that the data is missed out, depending on the query? Or would the
merging not be allowed to take place?
Would we want to define data slices for all partitions, except for the
current month ?
Or would we want to define a filter, and rather than a build and merge, do
an incremental build against the cube ?
We are really looking for the quickest load time possible, while having
correct data of course.
Thanks in advance for any advice and help.Is this for AS 2000 or 2005? In 2000, slices that you defined are used for
querying. In 2005, slices for MOLAP partitions are detected automatically.
For ROLAP partitions the slice you specify will be used to eliminate
partitions from queries.
But besides that, only your latest partition would have "incorrect" data,
right? So just don't set a slice for that last partition -- it would mean
that this partition would be unnecessarily scanned for some queries, but it
should be a fairly small partition anyway...
Thanks,
Akshai
--
This posting is provided "AS IS" with no warranties, and confers no rights
Please do not send email directly to this alias. This alias is for newsgroup
purposes only.
"Al" <Al@.discussions.microsoft.com> wrote in message
news:370E22EC-6304-49F2-B01D-812B37B2E1EF@.microsoft.com...
> Hi,
> we have cubes partitioned on the time dimension. The majority of data that
> comes in if for the last 4 days or so. The new data is loaded into a
> seperate
> fact table, and loaded into a partition. This partition is then merged
> into
> the current month partition.
> However, we can have data that comes in which is a month or longer old. So
> would be merged into the incorrect partition. And would remain there until
> a
> full reprocess of the Fact tables are cube partitiions.
> I understand that when data slices are defined for a partition, that
> AS2000
> will only visit the partitions that contain the relevent data. What
> happens
> if we use data slices in our situation?
> If the data is in the incorrect partition, as a result of merging, is it
> possible that the data is missed out, depending on the query? Or would the
> merging not be allowed to take place?
> Would we want to define data slices for all partitions, except for the
> current month ?
> Or would we want to define a filter, and rather than a build and merge, do
> an incremental build against the cube ?
> We are really looking for the quickest load time possible, while having
> correct data of course.
> Thanks in advance for any advice and help.
>sql
Saturday, February 25, 2012
Data Mart rollbacks
How have folks been managing rollbacks on failures inside SSIS when populating data marts?
For example - we have a seperate package for each dimension table, then a master Fact table update. If one of the dimension table updates fails - how have you rolled back the previous changes in the tables updated prior to the failure - or if the Fact tabel package fails - how do you manage rollback in all the dimension tables?
My first thought was using the Audit table information to determine which tables needed rolled back.
Hello Joe,
What about putting the Tasks (Execute Package tasks) in a transaction?
http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
> How have folks been managing rollbacks on failures inside SSIS when
> populating data marts?
> For example - we have a seperate package for each dimension table,
> then a master Fact table update. If one of the dimension table
> updates fails - how have you rolled back the previous changes in the
> tables updated prior to the failure - or if the Fact tabel package
> fails - how do you manage rollback in all the dimension tables?
> My first thought was using the Audit table information to determine
> which tables needed rolled back.
>
|||Is this what your team would implement?
"Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
news:885683c261f8c97ff6e74166f0@.news.microsoft.com ...
> Hello Joe,
> What about putting the Tasks (Execute Package tasks) in a transaction?
> http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
>
> --
> Allan Mitchell
> http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
> http://www.konesans.com
>
>
|||Hello Joe,
Yes. I would be looking to put things inside of transactions. I may logically
split things up but yes transactions would be the way for me
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
[vbcol=seagreen]
> Is this what your team would implement?
> "Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
> news:885683c261f8c97ff6e74166f0@.news.microsoft.com ...
|||the easy way:
backup the DB, execute the ETLs, restore the DB in case of a problem...
in fact, its a big recommendation to always backup first, so there is no overhead here.
"Joe" <hortoristic@.gmail.dot.com> wrote in message news:D6985AF3-799E-4B96-8A87-7A823C9C1FC2@.microsoft.com...
How have folks been managing rollbacks on failures inside SSIS when populating data marts?
For example - we have a seperate package for each dimension table, then a master Fact table update. If one of the dimension table updates fails - how have you rolled back the previous changes in the tables updated prior to the failure - or if the Fact tabel package fails - how do you manage rollback in all the dimension tables?
My first thought was using the Audit table information to determine which tables needed rolled back.
For example - we have a seperate package for each dimension table, then a master Fact table update. If one of the dimension table updates fails - how have you rolled back the previous changes in the tables updated prior to the failure - or if the Fact tabel package fails - how do you manage rollback in all the dimension tables?
My first thought was using the Audit table information to determine which tables needed rolled back.
Hello Joe,
What about putting the Tasks (Execute Package tasks) in a transaction?
http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
> How have folks been managing rollbacks on failures inside SSIS when
> populating data marts?
> For example - we have a seperate package for each dimension table,
> then a master Fact table update. If one of the dimension table
> updates fails - how have you rolled back the previous changes in the
> tables updated prior to the failure - or if the Fact tabel package
> fails - how do you manage rollback in all the dimension tables?
> My first thought was using the Audit table information to determine
> which tables needed rolled back.
>
|||Is this what your team would implement?
"Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
news:885683c261f8c97ff6e74166f0@.news.microsoft.com ...
> Hello Joe,
> What about putting the Tasks (Execute Package tasks) in a transaction?
> http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
>
> --
> Allan Mitchell
> http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
> http://www.konesans.com
>
>
|||Hello Joe,
Yes. I would be looking to put things inside of transactions. I may logically
split things up but yes transactions would be the way for me
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
[vbcol=seagreen]
> Is this what your team would implement?
> "Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
> news:885683c261f8c97ff6e74166f0@.news.microsoft.com ...
|||the easy way:
backup the DB, execute the ETLs, restore the DB in case of a problem...
in fact, its a big recommendation to always backup first, so there is no overhead here.
"Joe" <hortoristic@.gmail.dot.com> wrote in message news:D6985AF3-799E-4B96-8A87-7A823C9C1FC2@.microsoft.com...
How have folks been managing rollbacks on failures inside SSIS when populating data marts?
For example - we have a seperate package for each dimension table, then a master Fact table update. If one of the dimension table updates fails - how have you rolled back the previous changes in the tables updated prior to the failure - or if the Fact tabel package fails - how do you manage rollback in all the dimension tables?
My first thought was using the Audit table information to determine which tables needed rolled back.
Data Mart rollbacks
How have folks been managing rollbacks on failures inside SSIS when populati
ng data marts?
For example - we have a seperate package for each dimension table, then a ma
ster Fact table update. If one of the dimension table updates fails - how h
ave you rolled back the previous changes in the tables updated prior to the
failure - or if the Fact tabel package fails - how do you manage rollback in
all the dimension tables?
My first thought was using the Audit table information to determine which ta
bles needed rolled back.Hello Joe,
What about putting the Tasks (Execute Package tasks) in a transaction?
http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
> How have folks been managing rollbacks on failures inside SSIS when
> populating data marts?
> For example - we have a seperate package for each dimension table,
> then a master Fact table update. If one of the dimension table
> updates fails - how have you rolled back the previous changes in the
> tables updated prior to the failure - or if the Fact tabel package
> fails - how do you manage rollback in all the dimension tables?
> My first thought was using the Audit table information to determine
> which tables needed rolled back.
>|||Is this what your team would implement?
"Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
news:885683c261f8c97ff6e74166f0@.news.microsoft.com...
> Hello Joe,
> What about putting the Tasks (Execute Package tasks) in a transaction?
> http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
>
> --
> Allan Mitchell
> http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
> http://www.konesans.com
>
>|||Hello Joe,
Yes. I would be looking to put things inside of transactions. I may logica
lly
split things up but yes transactions would be the way for me
--
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
[vbcol=seagreen]
> Is this what your team would implement?
> "Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
> news:885683c261f8c97ff6e74166f0@.news.microsoft.com...
>|||the easy way:
backup the DB, execute the ETLs, restore the DB in case of a problem...
in fact, its a big recommendation to always backup first, so there is no ove
rhead here.
"Joe" <hortoristic@.gmail.dot.com> wrote in message news:D6985AF3-799E-4B96-8
A87-7A823C9C1FC2@.microsoft.com...
How have folks been managing rollbacks on failures inside SSIS when populati
ng data marts?
For example - we have a seperate package for each dimension table, then a ma
ster Fact table update. If one of the dimension table updates fails - how h
ave you rolled back the previous changes in the tables updated prior to the
failure - or if the Fact tabel package fails - how do you manage rollback in
all the dimension tables?
My first thought was using the Audit table information to determine which ta
bles needed rolled back.
ng data marts?
For example - we have a seperate package for each dimension table, then a ma
ster Fact table update. If one of the dimension table updates fails - how h
ave you rolled back the previous changes in the tables updated prior to the
failure - or if the Fact tabel package fails - how do you manage rollback in
all the dimension tables?
My first thought was using the Audit table information to determine which ta
bles needed rolled back.Hello Joe,
What about putting the Tasks (Execute Package tasks) in a transaction?
http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
> How have folks been managing rollbacks on failures inside SSIS when
> populating data marts?
> For example - we have a seperate package for each dimension table,
> then a master Fact table update. If one of the dimension table
> updates fails - how have you rolled back the previous changes in the
> tables updated prior to the failure - or if the Fact tabel package
> fails - how do you manage rollback in all the dimension tables?
> My first thought was using the Audit table information to determine
> which tables needed rolled back.
>|||Is this what your team would implement?
"Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
news:885683c261f8c97ff6e74166f0@.news.microsoft.com...
> Hello Joe,
> What about putting the Tasks (Execute Package tasks) in a transaction?
> http://msdn2.microsoft.com/en-us/library/ms137690(SQL.90).aspx
>
> --
> Allan Mitchell
> http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
> http://www.konesans.com
>
>|||Hello Joe,
Yes. I would be looking to put things inside of transactions. I may logica
lly
split things up but yes transactions would be the way for me
--
Allan Mitchell
http://wiki.sqlis.com | http://www.sqlis.com | http://www.sqldts.com |
http://www.konesans.com
[vbcol=seagreen]
> Is this what your team would implement?
> "Allan Mitchell" <allan@.no-spam.sqldts.com> wrote in message
> news:885683c261f8c97ff6e74166f0@.news.microsoft.com...
>|||the easy way:
backup the DB, execute the ETLs, restore the DB in case of a problem...
in fact, its a big recommendation to always backup first, so there is no ove
rhead here.
"Joe" <hortoristic@.gmail.dot.com> wrote in message news:D6985AF3-799E-4B96-8
A87-7A823C9C1FC2@.microsoft.com...
How have folks been managing rollbacks on failures inside SSIS when populati
ng data marts?
For example - we have a seperate package for each dimension table, then a ma
ster Fact table update. If one of the dimension table updates fails - how h
ave you rolled back the previous changes in the tables updated prior to the
failure - or if the Fact tabel package fails - how do you manage rollback in
all the dimension tables?
My first thought was using the Audit table information to determine which ta
bles needed rolled back.
Subscribe to:
Comments (Atom)