C# Class DataMigration.FeedProcessor

There's an instance of FeedProcessor for each DataSource in the Plan. It uses the DataSourcePlan and the contained FeedFilePlans to figure out what files to process, which parsers to use, which Stager to use, etc.

Inheritance: IFeedProcessor

ファイルを表示 Open project: scoutmedia/DataMigration

Public Methods

Method	Description
AddToBadRowsDataTable ( DataTable badRowDataTable, string dataSourceCode, BadRowException brex, IList rowData ) : void	Rows are processed in batches, and good and bad rows collected. This function add a single bad row to a datatable which is comprised of all the bad rows for a single batch.
InvokePostRowProcessor ( int loadNumber, IList row, string>.IDictionary rowInProgress, IMethodMap map ) : void	This function is called for PostRowProcessor methods specified in the FeedFileMap.
PostProcess ( int loadNum, IReadyToUseSubPlan readyToUseSubPlan ) : void	PostProcess is the final stage of processing, where business rules are applied against the data placed in staging by the Process() endpoint, and resultant rows inserted, modified, and/or deleted at the final location. The load number to perform post processing on. A ReadyToUseSubPlan (can be null) that may have additional PostProcessors to be run first.
Process ( ) : int	This entry point implements the first stage of processing for a datasource. The feed files have already been put in place (as a precondition), and this method will then parse the feed files, transform them as described by their respective transformMaps, and call the DataStager for this FeedProcessor to stage the intermediate results (i.e. store them for further processing by the PostProcessing entry point). The loadnumber that was generated as part of the processing. It is returned so it can be passed on to the PostProcessors.
ProcessPostRowProcessors ( int loadNumber, IList parsedInputRow, string>.IDictionary transformedRow, FeedFilePlan plan ) : void	Processes the post-row processor methods. These are methods that optionally run subsequent to a row successfully processing, that generate some desired side-effect.
RemoveDataSourceSpecificExistences ( IEnumerable existenceKeys ) : void	Removes the data source specific existences.
RemoveDataSourceSpecificLookups ( IEnumerable lookupKeys ) : void	Removes the data source specific lookups.

Private Methods

Method	Description
AcquireParser ( FeedFilePlan plan ) : IParser	Helper which acquires the parser either from the description in the FeedFilePlan (it favors this one), or if that's missing, from the ReadyToUseSubPlan.
AcquirePostPostProcessors ( IReadyToUseSubPlan readyToUseSubPlan ) : void	The DataSourcePlan has a set of PostProcessors, identified by fully-qualified assembly name. If there is a readyToUseSubPlan, and it has postprocessors, it will add those first. A ReadyToUsePlan (possibly null) they specify processors to be fired first.
AddDataSourceSpecificExistences ( ) : IEnumerable	Adds the data source specific existence Objects. These are described in the DataSourcePlan. They will be instantiated, and then later removed from the global set of Existence objects when processing this datasource is complete.
AddDataSourceSpecificLookups ( ) : IEnumerable	Adds the data source specific lookups. These are also described in the DataSourcePlan. They will be instantiated, and then later removed from the global set of Lookup objects when processing this datasource is complete.
AddToBadOrWarnRowsDataTable ( DataTable dataTable, string dataSourceCode, int loadNumber, int rowNumber, string destinationColumn, string reason, string foreignId, IList rowData ) : void	Helper function for adding rows to data table that takes the individual arguments. This makes it possible to do this without a BadRowException being present, as in the case of LitmusTests.
AddToWarnRowsDataTable ( DataTable warnRowsDataTable, string dataSourceCode, int loadNumber, int rowNumber, string destinationColumn, IList warnings, string foreignId, IList rowData ) : void	Helper function that adds all the warning rows for a given input row to the WarnRows datatable.
AssertPasswordSanity ( string>.IDictionary outRow, int lineNumber ) : void	Just a check for a mistake that has been made for FFT. Makes sure that, if we have calculated a PasswordImportState of "1", then there needs to be a PasswordHash.
FormXmlDocumentFromRowData ( IList rowData ) : string	Produces a simple xml object from the row Data. The reason to do it this way is that the BadRows table has to support any feed, with any schema. Instead of using a blob or nvarchar(max) column, the Xml data type is queryable.
GetFeedStager ( ) : IFeedStager	Gets the feed stager from either the readyToUsePlan or instantiates it through reflection from the main Plan.
InstantiatePostProcessor ( PostProcessorDescriptor postProcessorDescriptor ) : IPostProcessor	Instantiates a single PostProcessor.
PerformLitmusTest ( IList row, IMethodMap map ) : bool	This function is called for litmustest functions specified in the FeedFileMap.
ProcessLitmusTests ( IList row, FeedFilePlan plan ) : string	Processes the litmus tests.
ProcessWarningTests ( IList row, FeedFilePlan plan ) : List	Processes the warning tests. Warning Tests are litmus tests, the only difference being that the handling of a failure is different. Warning tests cause the row to be copied to the WarnRows table, but we still process the row.
RunPostProcessors ( int loadNum, IEnumerable postProcessors ) : void
SetUpDataTable ( FeedFilePlan feedFilePlan ) : DataTable	Creates and sets up the DataTable with column information.
SetupBadOrWarnRowsDataTable ( ) : DataTable	Instantiates and builds the column definitions for the bad rows DataTable.

Method Details

AddToBadRowsDataTable() public method

Rows are processed in batches, and good and bad rows collected. This function add a single bad row to a datatable which is comprised of all the bad rows for a single batch.

public AddToBadRowsDataTable ( DataTable badRowDataTable, string dataSourceCode, BadRowException brex, IList rowData ) : void
badRowDataTable	System.Data.DataTable	The data table to add to.
dataSourceCode	string	The DataSourceCode.
brex	BadRowException	The BadRowException that has contextual information.
rowData	IList	The raw parsed row data.
return	void

InvokePostRowProcessor() public method

This function is called for PostRowProcessor methods specified in the FeedFileMap.

public InvokePostRowProcessor ( int loadNumber, IList row, string>.IDictionary rowInProgress, IMethodMap map ) : void
loadNumber	int
row	IList	The raw data row.
rowInProgress	string>.IDictionary	The completed output row.
map	IMethodMap	An IMethodMap that describes how to perform the PostRowProcessor. ///
return	void

PostProcess() public method

PostProcess is the final stage of processing, where business rules are applied against the data placed in staging by the Process() endpoint, and resultant rows inserted, modified, and/or deleted at the final location. The load number to perform post processing on. A ReadyToUseSubPlan (can be null) that may have additional PostProcessors to be run first.

public PostProcess ( int loadNum, IReadyToUseSubPlan readyToUseSubPlan ) : void
loadNum	int
readyToUseSubPlan	IReadyToUseSubPlan
return	void

Process() public method

This entry point implements the first stage of processing for a datasource. The feed files have already been put in place (as a precondition), and this method will then parse the feed files, transform them as described by their respective transformMaps, and call the DataStager for this FeedProcessor to stage the intermediate results (i.e. store them for further processing by the PostProcessing entry point). The loadnumber that was generated as part of the processing. It is returned so it can be passed on to the PostProcessors.

public Process ( ) : int
return	int

ProcessPostRowProcessors() public method

Processes the post-row processor methods. These are methods that optionally run subsequent to a row successfully processing, that generate some desired side-effect.

public ProcessPostRowProcessors ( int loadNumber, IList parsedInputRow, string>.IDictionary transformedRow, FeedFilePlan plan ) : void
loadNumber	int	The load number.
parsedInputRow	IList	The raw row.
transformedRow	string>.IDictionary	The complete output row.
plan	FeedFilePlan	A FeedFilePlan with PostRowProcessorDescriptors.
return	void

RemoveDataSourceSpecificExistences() public method

Removes the data source specific existences.

public RemoveDataSourceSpecificExistences ( IEnumerable existenceKeys ) : void
existenceKeys	IEnumerable	Existence keys.
return	void

RemoveDataSourceSpecificLookups() public method

Removes the data source specific lookups.

public RemoveDataSourceSpecificLookups ( IEnumerable lookupKeys ) : void
lookupKeys	IEnumerable	Lookup keys.
return	void