SQL Server Business Intelligence: SSIS
Showing posts with label SSIS. Show all posts
Showing posts with label SSIS. Show all posts

Sunday, September 7, 2008

Database Snapshots and SSIS

clipped from blogs.conchango.com

Database Snapshots and SSIS

Did you think SQL Server 2005 database snapshots were really cool, but wondered how useful they were in practice? Building a Data Warehouse, here's an ideal use for them:

In my current project we're extracting data for the data warehouse from several SQL Server 2005 source system databases. Since we're extracting from multiple tables from each database, and since those tables continue to be populated during our extract, there is a definite risk that the extracted data is out of sync due to the extracts from different tables happens at slightly different times.

Database snapshots to the rescue

By first creating a snapshot of the source system database, and extracting from that static snapshot (instead of the original, changing database), these synchronization issues are avoided!

Even better, while the underlying database might be modified during the extract, the overhead for creating and using the snapshot is very small - it's proportional to the number of source system table pages changed during the extract.

Each source system database to be extracted has the following tasks configured:

Control Flow Image

1. The first Execute SQL Task creates the snapshot. Since no data is copied, this is a virtually instantaneous operation. At this stage the source system Connection Manager still points to the original database.

Creating the snapshot

2. The first Script Task switches the database ConnectionString from the original database to the newly created snapshot database. This avoids having to create and keep a second connection manager in sync with the original database.

Switch ConnectionString

3. The Sequence Container contains all the Data Flow tasks etc. needed to move the source system data (now guaranteed to be static during the extract) into the data warehouse. NB: the following precedence constraints are "On Completion", so even if the extract fails, the snapshot will be deleted.

4. The second Script Task switches the ConnectionString back to the original database, i.e. almost same code as 2 above, but change "Source_System_Snapshot_Extract" into "Source_System", and vice versa.

5. The final Execute SQL Task simply drops the snapshot, again an instantaneous operation.

Dropping the snapshot

Wrapping up

One thing to watch out for is when additional database files are created, either manually by the DBA or automatically due to the source system having a maximum file size set. The snapshot creation command must specify ALL database files, so either update the snapshot command manually when this happens, or extend the Script Tasks to automatically figure out the number and filenames required.

In summary, database snapshots and SSIS have proved to be a very useful combination for extracting consistent data into the data warehouse.

Published 06 March 2006 18:45 by Kristian.Wedberg
Get Clipmarks - The easiest way to email text, images and videos you find on the web.
Sent with Clipmarks

Friday, August 29, 2008

Step-By-Step: How to move SQL Server data efficiently with the Bulk Copy Program

Step-By-Step: How to move SQL Server data efficiently with the Bulk Copy Program


Takeaway: Learn to use the Bulk Copy Program (BCP), a command-line utility, to import and export large amounts of data in and out of SQL Server databases


The Bulk Copy Program (BCP) is a command-line utility that ships with Microsoft SQL Server. With BCP, you can import and export large amounts of data in and out of SQL Server databases quickly and easily. Any DBA who has utilized this functionality will agree that BCP is an essential tool.

BCP 101
You access the BCP utility from the command prompt. Here's the simple syntax:
bcp {dbtable | query} {in | out | queryout | format} datafile
[-n native type] [-c character type] [-S server name] [-U username]
[-P password] [-T trusted connection]

The command example I'm going to use starts with bcp followed by a fully qualified table name (database name, table or object owner, table or object name). For example, if you want to export the authors table, as part of the dbo group from the pubs database, you supply the full table name pubs.dbo.authors. Next, you use an in or out argument to specify whether you want BCP to copy data into or out of a database. You then specify the location of the datafile on your database server.

At this point, our BCP statement looks like this:
bcp pubs.dbo.authors out c:\temp\authors.bcp

Note
When using BCP, don't forget that the switches are case-sensitive. If you don't apply the right case, the BCP statement won't work correctly or will fail.

Next, we'll add some of the basic command-line options. The –n switch specifies native SQL Server format. The –c switch is used when formatting the file using Char as a datatype. If you use this switch, you can easily open your file with Excel. The –S switch enables you to add the server/instance name. The –U switch allows you to add the name of the login used to connect to SQL Server. The –P switch lets you add the password of the –U switch, and the –T switch is for establishing a trusted connection to your SQL Server. Figure A shows our completed BCP statement.

Figure A
Sample BCP out command


If we were importing data to the authors table, our BCP command would look like this:
bcp pubs.dbo.authors in c:\temp\authors.bcp -c -Sstevenw -Usa –P

That gives you an idea of the basics of the BCP utility, but there are plenty more switches at your disposal. In all, BCP supports 27 switches, which are shown in Listing A.
Listing A
bcp {dbtable | query} {in | out | queryout | format} datafile
  [-m maxerrors]            [-f formatfile]                  [-e errfile]
  [-F firstrow]             [-L lastrow]                     [-b batchsize]
  [-n native type]          [-c character type]              [-w wide character type]
  [-N keep non-text native] [-V file format version]          [-q quoted identifier]
  [-C code page specifier]  [-t field terminator]            [-r row terminator]
  [-i inputfile]            [-o outfile]                     [-a packetsize]
  [-S server name]          [-U username]                    [-P password]
  [-T trusted connection]   [-v version]                     [-R regional enable]
 [-k keep null values]     [-E keep identity values]         [-h "load hints"]


With this many switches available, I recommend that you play around with them to see which ones you might want to use. For example, the –e switch is handy because it will create an error file you can look at if your BCP command returns errors. Here is an example:
bcp pubs.dbo.authors out c:\temp\steventest.txt -n -Sstevenw -Usa -P
-eC:\temp\error.txt

For more information on how to use the various switches, see SQL Server Books Online.

BCP and database objects
Let's create a new database called pubs2 and use BCP to move all the data from the pubs database into it. First, we need to create the database. Then, we'll generate the SQL script, run it on pubs2, and export the data from pubs into pubs2.

Start by creating the database in SQL Server Enterprise Manager and name it pubs2. Next, select pubs2, right-click on it, and choose Generate SQL Scripts from the All Tasks menu. Now you are ready to set your scripting options. Click Show All and then select the Script All Objects check box, as shown in Figure B.

Figure B
The Generate SQL Scripts dialog box


Go to the Options tab and select all the check boxes under Table Scripting Options, as we've done in Figure C. You can preview your script from the General tab or click OK to create and save the script.

Figure C
Table Scripting Options


Note
By performing the above, we are making a complete replica of the Pubs database. We could also have scripted out only one table or multiple tables. You can make the choice, depending on the task at hand.

Now, open Query Analyzer and run the newly created script (Figure D).
http://img.com.com/i/tr/cms/contentPics/r00220020620wrr01_09.gif
Run the script in SQL Query Analyzer.
After the script runs, the database objects and indexes will have been successfully created (Figure E).

http://img.com.com/i/tr/cms/contentPics/r00220020620wrr01_10.gif
The database objects have been created in SQL Server Enterprise Manager

To export the authors table out of pubs and import it into pubs2, we'll use the BCP statements we learned earlier:
bcp pubs.dbo.authors out c:\temp\pubauthors.bcp –n –Sstevenw –Usa –P
bcp pubs2.dbo.authors in c:\temp\pubauthors.bcp –n –Sstevenw –Usa –P

Then, open the SQL Query Analyzer and run the following statements on the pubs2 and pubs databases:
Use pubs2
Select * from authors
Select count(*) from authors

Use pubs
Select * from authors
Select count(*) from authors

This allows you to see the full process of moving data come full circle.

Summary
Most DBAs today use DTS, Database Restore, and/or attaching and detaching databases to copy data from one server to another. All these tools will get the job done. But if you want to copy large amounts of SQL Server data quickly, BCP is still one of the best tools available. Once you get past the initial learning curve, you'll find that BCP offers a highly efficient way to copy data from one database to another.

Monday, August 4, 2008

Managing SSIS Service

Help JumpStart.TV to bring us more knowledge and Videos on SSIS.














If you liked that, join Jumpstart TV Inc., who has All Rights Reserved