Custom Script Destination - "wrote 0 rows" Problems

I have created a custom script destination using an article found on TechNet. The script executes fine and completes, but with one small problem. The last package execution log entry states "wrote 0 rows". This is a problem because I use this number to verify that all of the database records have been properly transferred.

I have monitored that database when the script is executing and have verified that the data is being transferred property.

Is there some function that I should be calling with each row prcessed in the custom script destination to "tally" the row count? This way the script component will display the proper row count.

Any suggestions would be appreciated.

You could increment a counter variable in the ProcessInputRow sub and then in the PostExecute sub, assign the value of that variable back to a package-level variable. Then you can, if desired, use an Execute SQL task to insert the contents of the package variable into an audit table. Of course, you could just use the package variable in an expression as well.

I assume you are using this method solely because you are using an ODBC destination? (Just checking)

|||

That is a good work around and I think I will implement something like that if I am unable to find a solution.

I still would like to know the "correct" fix on how to output the row count.

Thanks for the post and I will be sure to post any updates if I find anything out.

--Ryan

|||The way to update the row count in that informational message is to call IncrementPipelinePerfCounter on IDTSComponentMetadata90 with the DTS_PIPELINE_CTR_ROWSWRITTEN flag.|||

Ryan Teribery wrote:

I have created a custom script destination using an article found on TechNet. The script executes fine and completes, but with one small problem. The last package execution log entry states "wrote 0 rows". This is a problem because I use this number to verify that all of the database records have been properly transferred.

I have monitored that database when the script is executing and have verified that the data is being transferred property.

Is there some function that I should be calling with each row prcessed in the custom script destination to "tally" the row count? This way the script component will display the proper row count.

Any suggestions would be appreciated.

A custom script destination will not do this by default. The provided destination components have this functionality embedded within them and you would need to embed the same in the script component.

Use:

Me.ComponentMetadata.FireInformation(...)

in your script component to pass out the information that you require. You should use this code in the PostExecute() method.

Counting the number of rows shouldn't be a problem. Just increment a private variable within the ProcessInputRow() method.

Regards

Jamie

|||

Ryan Teribery wrote:

That is a good work around and I think I will implement something like that if I am unable to find a solution.

it is not a workaround. This is the correct way to do it.

Ryan Teribery wrote:

I still would like to know the "correct" fix on how to output the row count.

The technique that Phil and I have described IS the correct way to do this. Phil's differs slightly because he uses an Execute SQL Task to write a record where mine throws an event which gets caught by your log provider. I think my way is better but then, I'm biased (Phil won't mind me saying that)

|||

The embedded functionality in the provided transforms (e.g. OLE DB Dest adapter) is the use of the IncrementPipelinePerfCounter method I mentioned earlier. Here's how this works: when that method is called by your destination component, the data flow engine tracks the row count your component passed to it (so you still need to track the row count internall). When the data flow finishes its execution (i.e. after the Cleanup phase), the data flow engine will fire DTS_I_ROWS_WRITTEN with the row count on behalf of the component, resulting in the "Destination Component wrote X rows" message. Your component does not need to post its own message. Also note that the perf counters for rows written will be updated as well, which may be good or bad depending on what you want.

As mentioned by others, there are other ways to handle this. Hope this additional info will be useful in helping you decide how you need to implement this.

Custom Properties in External Metadata not writeable using the Advanced Editor

Hello,

I'm working on a custom dataflow destination component. It makes use of the External Metadata Collection. I also use Custom Properties with the external metadata collection.

When I open the destination component using the Advanced Editor, and select an External Metadata Collection and change the Custom Property it always changes back to the original value.

Additionally the method SetExternalMetadataColumnProperty never gets called.

Here is a little Test Component that surfaces the problem:

[DtsPipelineComponent(ComponentType=ComponentType.DestinationAdapter, DisplayName="Test Destination")]
public class Class1 : PipelineComponent
{
public override void ProvideComponentProperties()
{
base.ProvideComponentProperties();

ComponentMetaData.InputCollection.RemoveAll();
IDTSInput90 input = ComponentMetaData.InputCollection.New();
input.Name = "Name";
input.ExternalMetadataColumnCollection.IsUsed = true;

IDTSExternalMetadataColumn90 ext = input.ExternalMetadataColumnCollection.New();
ext.Name = "ExtName";
IDTSCustomProperty90 customProp = ext.CustomPropertyCollection.New();
customProp.Name = "Custom Property";
customProp.Value = "Hallo";
}

public override IDTSCustomProperty90 SetExternalMetadataColumnProperty(int iID, int iExternalMetadataColumnID, string strPropertyName, object oValue)
{
return base.SetExternalMetadataColumnProperty(iID, iExternalMetadataColumnID, strPropertyName, oValue);
}
}

Any ideas how to make that work?

Georg

Are you sure that your SetExternalMetadataColumnProperty method is getting called? I believe you must implement the functionality to set the property yourself rather than delegating to the base class.|||

No, actually part of problem is that it never gets called when I change the property.

The method is only implemented for debugging. If you set a breakpoint in it, it never gets called. As far as I understand is, you would override this method just to implement additional functionality when a custom property of an External Metadata Column is changed, eg for validation.

Custom Destination Component Logging - wrote 0 rows

I wrote a custom destination component. Everything works fine, except there is a logging message that is displayed that I cannot get rid of or correct. Here is the end of the output of a package containing my component:

Information: 0x40043009 at Data Flow Task, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x0 at Data Flow Task, MyDestination: Inserted 40315 rows into C:\temp\file.txt
Information: 0x4004300B at Data Flow Task, DTS.Pipeline: "component "MyDestination" (9)" wrote 0 rows.
SSIS package "Package.dtsx" finished: Success.

I inserted a custom information message that contains the correct number of rows written by the component. I would like to either get rid of the last message "... wrote 0 rows", or figure out what to set to put the correct number of rows into that message.

This message seems to happen in the Cleanup phase. It appears whether I override the Cleanup method of the Pipeline component and do nothing, or not. Any ideas?

public override void Cleanup()

{

ComponentMetaData.FireInformation(0, ComponentMetaData.Name,

"Inserted " + m_rowCount.ToString() + " rows into " + m_fileName,

"", 0, ref m_cancel);

base.Cleanup(); // or not

}

This message comes from the engine and you can not stop it from occuring. To set it correctly you need to call the IncrementPipelinePerfCounters(counter, difference) method on the IDTSComponentMetaData90 interface.

The counter values are:

RowsRead: 101

RowsWritten: 103

BlobBytesRead: 116

BlobBytesWritten: 118

The difference value is the amount to increment the counter by.

Thanks,

Matt

|||

Thanks. That did the trick.

Apparently there are constants defined for the counter types, but I'm not sure where they are.

Here's the official help page:

http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.dts.pipeline.wrapper.idtscomponentmetadata90.incrementpipelineperfcounter.aspx

|||

The constants for the counter types are listed in the Remarks section of the BOL page for which you pasted the link. Are those not what you were looking for?

-Doug

|||

Those constants work. I was hoping to be able to access the constants by an actual constant, instead of having to hard-code the number.

I would rather have code that looks like this:

ComponentMetaData.IncrementPipelinePerfCounter(DTS_PIPELINE_ROWS_WRITTEN, m_rowCount);

than this:

ComponentMetaData.IncrementPipelinePerfCounter(103, m_rowCount);

For now I just created a local constant with the above name and that works fine.

private const uint DTS_PIPELINE_ROWS_WRITTEN = 103;

Maybe those constants are exposed somewhere, but I could not figure out where.

|||

Excuse me for missing the point.

I suspect that these counter constants are defined in the native pipeline engine and not exposed in any of the managed classes...that would explain why this managed method expects an integer value and not an enum value. You could create your own enumeration for this purpose.

-Doug

Custom Destination Adapter

Hi All,

I have built a custom flat file destination adapter but it appears that the code is not working. When I debug the process I notice that the ProcessInput section is called multiple times. The first time it looks like everything is working, then it call it again and there is no inpout from the DTSInput90.

Why would it do this?

Thanks

Mike

Data moves down the pipeline in buffers. It is not one stream, or individual rows. Obviously buffers have a limited capacity, so you will get multiple buffers depending on the number of rows. The rows per buffer depends on the size of the row essentially.

So ProcessInput gets called once per buffer. This is why you cache buffer information in PreExecute, so that you get re-use. Querying the information can be quite expensive which is why it is cached to start with.

I'm not sure what you mean by "there is no inpout from the DTSInput90", can you explain?

|||Basically what I am saying is that the second time through there is no data. So if you want to capture all the data from the buffer and write it all out in a file how would you do that?|||

That doesn't sound particularly useful, but maybe how it works. Not sure I've ever stepped through to that degree. It should not cause you any problems however, but are you getting all the data you expect? Is there actually a problem?

|||

I figured it out. The detination component takes the input stream and creates a ZIP file with some delimiter that I have defined. I needed to create a global variable for the string and keep appending until then buffers were complete and then run through the ZIP code.

Thanks

Custom Destination Adapter

Hi All,

I have built a custom flat file destination adapter but it appears that the code is not working. When I debug the process I notice that the ProcessInput section is called multiple times. The first time it looks like everything is working, then it call it again and there is no inpout from the DTSInput90.

Why would it do this?

Thanks

Mike

Data moves down the pipeline in buffers. It is not one stream, or individual rows. Obviously buffers have a limited capacity, so you will get multiple buffers depending on the number of rows. The rows per buffer depends on the size of the row essentially.

So ProcessInput gets called once per buffer. This is why you cache buffer information in PreExecute, so that you get re-use. Querying the information can be quite expensive which is why it is cached to start with.

I'm not sure what you mean by "there is no inpout from the DTSInput90", can you explain?

|||Basically what I am saying is that the second time through there is no data. So if you want to capture all the data from the buffer and write it all out in a file how would you do that?|||

That doesn't sound particularly useful, but maybe how it works. Not sure I've ever stepped through to that degree. It should not cause you any problems however, but are you getting all the data you expect? Is there actually a problem?

|||

I figured it out. The detination component takes the input stream and creates a ZIP file with some delimiter that I have defined. I needed to create a global variable for the string and keep appending until then buffers were complete and then run through the ZIP code.

Thanks

Custom Data Flow Component Column Mapping Question

Hi,

I'm having my first go at developing a destination adapter which will send data to an update Web Service.

I've got some rather big gaps in my understanding. I've been following the various samples I've found on the net and have validated my mapping and picked up all the available column names and datatypes which are appearing in the Input and Output Properties tab of the Advanced Editor but I only have a tab for "Input Columns" and not "Column Mappings".

Which method defines the availble columns for the user to map?

Let me know if I haven't given enough information.

cheers

sorry - I had searched already but somehow missed this post which answered my question: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=239126&SiteID=1

I was having the same confusion about input and output, thinking that because i was creating a *destination* adapter, that I should be creating outputs not inputs [if that makes sense].

Custom Data Flow

To all the SSIS gurus,

I have a situation where I have multiple source tables, I need to populate the destination tables that have the same schema as the source ones. I dont want to do the repetitive task of creating a Dataflow for each source-destination load.

I want to create one custom dataflow component and loop through all my source tables and provide destination tables dynamically.

Is there a way to do that? Any custom data flow component out there?

Your ideas are greatly appreciated.

Regards,

ssiscoder wrote:

To all the SSIS gurus,

I have a situation where I have multiple source tables, I need to populate the destination tables that have the same schema as the source ones. I dont want to do the repetitive task of creating a Dataflow for each source-destination load.

I want to create one custom dataflow component and loop through all my source tables and provide destination tables dynamically.

Is there a way to do that? Any custom data flow component out there?

Your ideas are greatly appreciated.

Regards,

No you can't do this currently. Well, not with the out-of0the-box components anyway.

What you CAN do is build a package programatically in dotnet using the SSIS API. That dotnet code can be called from any place that can execute dotnet code. One of those places happens to be a SSIS package.

HTH.

-Jamie

|||

Thanks for the info Jamie!

Can you please post some articles here of how to go about doing the data flow part programitically?

Regards,

|||

ssiscoder wrote:

Thanks for the info Jamie!

Can you please post some articles here of how to go about doing the data flow part programitically?

Regards,

I can do better than that. here's a sample:

http://msdn2.microsoft.com/en-us/library/ms161541.aspx

-Jamie

|||

This might help as well: http://msdn2.microsoft.com/en-us/library/ms345167.aspx

-Jamie