Trigger SSIS package when files available in a Folder part#2

In continuation to my previous post

 Trigger SSIS package when files available in a Folder

Windows Service Approach

Open Microsoft VisualStudio 2010, select New Project and click on Windows Service.

image

I named the solution as FileWindowsService, right click on References and select ‘Add Reference…’

image

Under .NET tab select System.Configuration , System.Configuration.Install and System.management

  • System.Configuration namespace allows us to handling the configuration data.
  • System.Configuration.Install namespace allows us to write custom installers for our own components.
  • System.management namespace allows us to Provides access to a rich set of management information and management events about the system, devices, and applications instrumented to the Windows Management Instrumentation (WMI) infrastructure.

image

Press Ctrl+Shift+A to add Application Configuration File or else follow the figures to add configuration file through which we will the pass the path of the directory to watch.

image

image

Follow the below figure to add directory path in the configuration file.

<?xml version="1.0" encoding="utf-8" ?>

<configuration>

<appSettings>

<add key="WatchPath" value="F:\\DellmeSoon\\" />

</appSettings>

</configuration>

image

Now open Service1.cs[Design] and drag FileSystemWatcher from the Toolbox which is in Components group.

FileSystemWatcher component is to watch for changes in a specified directory. You can watch for changes in files and subdirectories of the specified directory. You can create a component to watch files on a local computer, a network drive, or a remote computer.

image

Right Click on fileSystemWatcher1 tool and select properties. Change the Name and the NotifyFilter properties to FileName, DirectoryName, Size, LastWrite, LastAccess, CreationTime, Security

If you want to include subdirectories of the source folder that enable that property.

image

To watch for changes in all files, set the Filter property to an empty string ("") or use wildcards ("*.*").To watch a specific file, set the Filter property to the file name or to watch only certain file formats set the Filter Property to file extension like “.txt” for text files. You can watch for changes in Attributes, the LastWrite date and time, or the Size of files or directories. This is done by setting the NotifyFilters property to one of the NotifyFilters values.

Add Microsoft.SqlServer.ManagedDTS reference and add a using statement for the Microsoft.SqlServer.Dts.Runtime namespace.

image

Add another using statement for System.Configuration

image

   1: using System;

   2: using System.Collections.Generic;

   3: using System.ComponentModel;

   4: using System.Data;

   5: using System.Diagnostics;

   6: using System.Linq;

   7: using System.ServiceProcess;

   8: using System.Text;

   9: using System.Configuration;

  10: using Microsoft.SqlServer.Dts.Runtime;

  11:  

  12: namespace FileWindowsService

  13: {

  14:     public partial class Service1 : ServiceBase

  15:     {

  16:         public Service1()

  17:         {

  18:             InitializeComponent();

  19:         }

  20:         protected override void OnStart(string[] args)

  21:         {

  22:             FileWindowService.Path = ConfigurationManager.AppSettings["WatchPath"];

  23:         }

  24:         protected override void OnStop() { }

  25:         private void FileWindowServiceOnChanged(object sender,System.IO.FileSystemEventArgs e) { }

  26:         private void FileWindowServiceOnCreated(object sender,System.IO.FileSystemEventArgs e)

  27:         {

  28:             Microsoft.SqlServer.Dts.Runtime.Application app = new Microsoft.SqlServer.Dts.Runtime.Application(); 

  29:             Package createInvoice = app.LoadFromSqlServer("\\MyPackage", "MyProducitonServer", null, null, null); 

  30:             //createInvoice.Variables["PeriodID"].Value = VariableValue;

  31:             DTSExecResult result = createInvoice.Execute(); 

  32:         }

  33:         private void FileWindowServiceOnDeleted(object sender,System.IO.FileSystemEventArgs e) { }

  34:         private void FileWindowServiceOnRenamed(object sender, System.IO.RenamedEventArgs e) { }

  35:      }

  36:   }

Above C# code will triggers the events associated with file Created, Changed, Deleted and Renamed.

I have added code to call a SSIS package when the file is placed in the given source folder.

We need to map the methods in C# code to the events in the design pane.

In Services1.cs[Design] click on FileWindowService and select properties, click on events and map the respective methods to the event. Like FileWindowServiceonCreated to the Created event.

image

We have done with the code, now we need to create windows installer class to allow the project to be compiled as a Windows service.

Now add Installer Class (Follow the process of adding Configuration file)

image

Drag two components ServiceInstaller and ServiceProcessInstaller, if these components are located in the tool box, tight click on the tool box and the “choose items…” in .NET Framework Components

image

Select the properties of ServiceInstaller1 and set the ServiceName property to FileWindowService

image

Select the properties of serviceProcessInstaller1 and set the Account property to LocalSystem, which enables the service to be run as a local system account.

image

Now, simply Build FileWindowService (from the Build menu), and all necessary installer files will be created within the project directory. To install the newly created service, you must use the .NET Framework InstallUtil program.

.exe will be generated in the bin\debug of the project path.

image

Let me know the issue which comes to your way while follow the above process

Thanks a lot for visiting the blog…Smile

Trigger SSIS package when files available in a Folder

Recently I have goon through a scenario where my client needs to run the SSIS package when the files placed in folder. To run the package automatically when the file exists we have an external tool called FileWatcherTask which is a great user friendly (be aware that FileWatcherTask can cause memory leak and it’s been discussed in the SSIS community. It sounds like you can avoid using FWT altogether) but we can find similar task in Control Flow Tool box, which is WMI Event watcher task.

Different Approaches to run the Package when file exists.

  • Use the WMI Event Watcher task to trigger the SSIS packages as files arrive.
  • Create a Windows service that uses WMI to detect file arrival and launch packages.

Let’s discuss each of the available approach.

In this post I would like to show how to WMI Event Watcher task and other approaches to run SSIS package automatically when a file exists.

WMI Event Watcher Task:

WMI Event Watcher Task makes SSIS to wait for and respond to certain WMI events that occur in the operating System.

Scenarios where we can use WMI Event Watcher Task:

  • To watch a directory for a certain file to be written.
  • To watch for the CPU to be free.
  • Wait for a given service to start.
  • Wait for the memory of a server to reach a certain level before executing the rest of the package or before transferring files to the server.

The task uses a subset of SQL language which is WQL language; it allows us to view information through WMI.

My example is moving a file from Source folder to Archive folder with a FileSystem Task, WMI Event Watcher will pass the execution to FileSystem task when a file placed in Source Folder.

I took two variables to pass the Source Folder path dynamically.

1. User:: Quote = “

2. User::SourcePath = F:\\\\DellMeSoon\

Drag WMI Event Watcher task to the Control Flow Design pane, double click on the task.

In WMI Event Watcher Task Editor, we will need to create WMI Connection Manager, leave the Server name and Namespace options and select the check box to Use Windows Authentication

Click on Test button to test the Connection.

image

Select WqlQuerySourceType as DirectInput and for

WqlQuerySource keep . (dot)

image

Under Expressions tab and select WQLQuerySource property and use the following WQL Query as Expression.

Expression:

"SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE TargetInstance ISA "+@[User::Quote]+ "CIM_DirectoryContainsFile" + @[User::Quote]+"

AND TargetInstance.GroupComponent= "+@[User::Quote]+ "Win32_Directory.Name=\\" + @[User::Quote]+ @[User::SourcePath] + @[User::Quote]+ @[User::Quote]

There are three distinct classes:

  • __InstanceCreationEvent class, representing creation of a new object of a WMI class,
  • __InstanceDeletionEvent class, representing deletion of an existing object of a WMI class,
  • __InstanceModificationEvent class, representing modification to an existing object of a WMI class.

These classes are designed with the intention of being paired up with another WMI class, providing the ability to monitor changes to its objects.

WITHIN keyword set the interval between the event taking place and its notification being delivered to the WMI-based application (WMI Event Task, in this case).

WHERE clause includes the ISA keyword followed by the name of a monitored class.

image

Other properties includes:

ActionAtEvent: Specify whether the WMI event logs the event and initiates an SSIS action, or only logs the event.

AfterEvent: Specify whether the task succeeds or fails after it receives the WMI event, or if the task continues watching for the event to occur again.

ActionAtTimeout: Specify whether the task logs a WMI query time-out and initiates an SSIS event in response, or only logs the time-out.

AfterTimeout: Specify whether the task succeeds or fails in response to a time-out, or if the task continues watching for another time-out to recur.

NumberOfEvents: To specify the number of events to watch for.

Timeout: Specify the number of seconds to wait for the event to occur. A value of 0 means that no time-out is in effect.

Drag FileSystem task to move the File from Source folder to the Archive folder once available.

image

Now, execute the package. WMI Event watcher task will start watching the Source Folder till the file available. The task will be in execution mode and watch the source folder for every nth time which we gave in the WQL Query.

image

When the file is placed in the source folder WMI Event Watcher task will pass the execution to the next connected task.

image

Keep the package in Infinity loop for continuous watching of Source Folder.

Lets discuss about Windows Service Approach in my next post.

Execute SSRS report from SSIS package

Hi, Today my post is on ‘How to execute and render SSRS report (Excel format) from SSIS package’.

Am assuming a SSRS report (Employee Details) which accepts ‘Department’ as parameter, please have a glance at the Employee table and Report generated for particular department.

image

clip_image002

I’ll explain you how am going to execute the ‘EmployeeReport’ report from SSIS package.

Note:

@parameter1’ is the parameter which am passing to the report, please go through the ‘Dataset properties’ figure of SSRS, we need to use the same name in the SSIS package.

clip_image012

Create a SSIS package and Drag Script task into the design pane and before editing change the ‘Script language’ property to ‘Microsoft Visual Basic 2008

clip_image004

Variables in SSIS package:

  • User::FileExtension - To generate report with given File extension.
  • User::FileType - File Format name of given file extension.
  • User::FullFilePath - To generate file in given file path.
  • User::Filename - To dynamically name the generated file.
  • User::Parameter1 - To pass parameter to the SSRS report.

Right click on  ‘Script task’ and select ‘Edit’. Select FileExtension, FileType, FullFilepath, parameter1 as ‘ReadOnlyVariables’

clip_image006

After selecting ‘ReadOnlyVariables’, click on ‘Edit Script’.

image

Web Reference:

We need to add ‘ReportExecutionService’  as a web reference, right click on the class name and select ‘Add Web reference’ property.

clip_image008

Paste ‘http://localhost/reportserver/ReportExecution2005.asmx?WSDL’ in the URL box and click on GO, we’ll get the description regarding ‘ReportExecutionService’ and click on ‘Add reference ‘ after giving some name to the Web reference(‘localhost’ in my example).

clip_image010

Script task Code (Microsoft Visual Basic 2008)

We need to import ‘System.IO’ class to deal with File stream objects, please go through below figure for the Script task code.

image

‘.LoadReport("/EmployeeReport/EmployeReport", Nothing)’ : ‘/EmployeeReport/EmployeReport’ is where the Report stored in Report Manager.

Report will be generated in the given File path with ‘Report name’ as the ‘Excel sheet name’

clip_image016

Let me know your comments on my Post.

Using OUTPUT Clause in DML statements.

The OUTPUT clause returns a copy of the data that you’ve inserted into or deleted from your tables. You can return that data to a temporary or permanent table, a table variable, or to the processing application that’s calls the DML statement. You can then use that data for such purposes as archiving, confirmation messaging, or other application requirements.

Both SQL Server 2005 and 2008 let you add an OUTPUT clause to INSERT, UPDATE, or DELETE statements. SQL Server 2008 also lets you add the clause to MERGE statements.

The OUTPUT clause takes two basic forms:

i. OUTPUT (want to return the data to the calling application)

ii. OUTPUT INTO (to return the data to a table or a table variable.)

In the below example I just created a table called ‘Test1’ and I used a table variable called ‘@TempTest1’ to show the computed records.

image

OUTPUT Clause in an INSERT Statement:

We can use OUTPUT clause while inserting data into a table, to return a copy of the data that’s been inserted into the table.

INSERT INTO dbo.Test1

OUTPUT INSERTED.sno

                    ,INSERTED.ename

                    ,INSERTED.eloc

INTO     @TempTest1

VALUES (1,’peter’,’MI’)

image

OUTPUT Clause in an UPDATE Statement:

I have just inserted a record into ‘Test1’ table, now I want to update the ename value of that record(from ‘Peter’ to ‘John Peter’). We can get both the previous value and updated value if we use OUTPUT clause while updating the ename value.

UPDATE     dbo.Test1

SET ename=’John Peter’

OUTPUT  INSERTED.sno

                     ,DELETED.ename

                     ,INSERTED.ename

                    ,INSERTED.eloc

INTO @TempTest_updation

WHERE sno=1

image

OUTPUT Clause in an DELETE Statement:

We can use OUTPUT clause in a DELETE statement to archive the records from a table into Archive table. The OUTPUT clause will insert deleted records in archive table.

DELETE dbo.Test1

OUTPUT DELETED.sno

,DELETED.ename

,DELETED.eloc

INTO @TempTest_Deletion

image

The OUTPUT clause is not supported in the following statements:

  1. DML statements that reference local partitioned views, distributed partitioned views, or remote tables.
  2. INSERT statements that contain an EXECUTE statement.
  3. Full-text predicates are not allowed in the OUTPUT clause when the database compatibility level is set to 100.
  4. The OUTPUT INTO clause cannot be used to insert into a view, or rowset function.
  5. A user-defined function cannot be created if it contains an OUTPUT INTO clause that has a table as its target.

The OUTPUT clause cannot contain the following references:

  1. · Sub queries or user-defined functions that perform user or system data access, or are assumed to perform such access. User-defined functions are assumed to perform data access if they are not schema-bound.
  2. · A column from a view or inline table-valued function when that column is defined by one of the following methods:
  • A sub query.
  • A user-defined function that performs user or system data access, or is assumed to perform such access.
  • A computed column that contains a user-defined function that performs user or system data access in its definition.

Number of character occurrences in String

To find number of character occurrences in a String.

image

To test, execute the function and ..

image

Convert alphanumeric to numeric

Below script is to convert alpha numeric mobile numbers to numeric

image

here I have given mobile number as ‘1-800-MSFT’

Result is:

image

Generate insert statements of records in a table

We may need to transfer records from Development to Production, we have many ways to do that, let me show how to generate ‘Insert statements’ using StoredProcedure for the required SQL Server table from which we need to transfer data.

download ‘GenerateInsertStatements‘ script to generate ‘Insert Statement’ of a SQL Server Table

Let me show how it works…

Script need two parameters ‘table name’ and ‘schema name’, if we submit those two parameters it will generate ‘Insert statements’ for the given table.

Another case here is to consider IDENTITY column while generating insert data script. Let’s revise the above example by considering Id as identity column.

I have attached the procedure, just try using the procedure in your free time…

Rename DB Constraint

image

Know about costliest query running right now

  • By Memory usage

image

  • By Query cost.

image

Scripts to find details about ‘logins’ & ‘Roles’

Script to find server level logins and role assigned

image

 

Script to find database users and roles assigned

image

 

 

 

Script to find Object level permission for user databases

image

Follow

Get every new post delivered to your Inbox.